| blob | 3842b6481c309ec5138576e838e46ea2882ce521 |
1 .\" $File: magic.man,v 1.39 2007/11/08 00:31:37 christos Exp $
2 .Dd January 10, 2007
3 .Dt MAGIC __FSECTION__
4 .Os
5 .\" install as magic.4 on USG, magic.5 on V7 or Berkeley systems.
6 .Sh NAME
7 .Nm magic
8 .Nd file command's magic number file
9 .Sh DESCRIPTION
10 This manual page documents the format of the magic file as
11 used by the
12 .Xr file __CSECTION__
13 command, version __VERSION__.
14 The
15 .Xr file __CSECTION__
16 command identifies the type of a file using,
17 among other tests,
18 a test for whether the file begins with a certain
19 .Dq "magic number" .
20 The file
21 .Pa __MAGIC__
22 specifies what magic numbers are to be tested for,
23 what message to print if a particular magic number is found,
24 and additional information to extract from the file.
25 .Pp
26 Each line of the file specifies a test to be performed.
27 A test compares the data starting at a particular offset
28 in the file with a 1-byte, 2-byte, or 4-byte numeric value or
29 a string.
30 If the test succeeds, a message is printed.
31 The line consists of the following fields:
32 .Bl -tag -width ".Dv message"
33 .It Dv offset
34 A number specifying the offset, in bytes, into the file of the data
35 which is to be tested.
36 .It Dv type
37 The type of the data to be tested.
38 The possible values are:
39 .Bl -tag -width ".Dv lestring16"
40 .It Dv byte
41 A one-byte value.
42 .It Dv short
43 A two-byte value (on most systems) in this machine's native byte order.
44 .It Dv long
45 A four-byte value (on most systems) in this machine's native byte order.
46 .It Dv quad
47 An eight-byte value (on most systems) in this machine's native byte order.
48 .It Dv float
49 A 32-bit (on most systems) single precision IEEE floating point number in this machine's native byte order.
50 .It Dv double
51 A 64-bit (on most systems) double precision IEEE floating point number in this machine's native byte order.
52 .It Dv string
53 A string of bytes.
54 The string type specification can be optionally followed
55 by /[Bbc]*.
56 The
57 .Dq B
58 flag compacts whitespace in the target, which must
59 contain at least one whitespace character.
60 If the magic has
61 .Dv n
62 consecutive blanks, the target needs at least
63 .Dv n
64 consecutive blanks to match.
65 The
66 .Dq b
67 flag treats every blank in the target as an optional blank.
68 Finally the
69 .Dq c
70 flag, specifies case insensitive matching: lowercase
71 characters in the magic match both lower and upper case characters in the
72 targer, whereas upper case characters in the magic, only much uppercase
73 characters in the target.
74 .It Dv pstring
75 A pascal style string where the first byte is interpreted as the an
76 unsigned length.
77 The string is not NUL terminated.
78 .It Dv date
79 A four-byte value interpreted as a UNIX date.
80 .It Dv qdate
81 A eight-byte value interpreted as a UNIX date.
82 .It Dv ldate
83 A four-byte value interpreted as a UNIX-style date, but interpreted as
84 local time rather than UTC.
85 .It Dv qldate
86 An eight-byte value interpreted as a UNIX-style date, but interpreted as
87 local time rather than UTC.
88 .It Dv beshort
89 A two-byte value (on most systems) in big-endian byte order.
90 .It Dv belong
91 A four-byte value (on most systems) in big-endian byte order.
92 .It Dv bequad
93 An eight-byte value (on most systems) in big-endian byte order.
94 .It Dv befloat
95 A 32-bit (on most systems) single precision IEEE floating point number in big-endian byte order.
96 .It Dv bedouble
97 A 64-bit (on most systems) double precision IEEE floating point number in big-endian byte order.
98 .It Dv bedate
99 A four-byte value (on most systems) in big-endian byte order,
100 interpreted as a Unix date.
101 .It Dv beqdate
102 An eight-byte value (on most systems) in big-endian byte order,
103 interpreted as a Unix date.
104 .It Dv beldate
105 A four-byte value (on most systems) in big-endian byte order,
106 interpreted as a UNIX-style date, but interpreted as local time rather
107 than UTC.
108 .It Dv beqldate
109 An eight-byte value (on most systems) in big-endian byte order,
110 interpreted as a UNIX-style date, but interpreted as local time rather
111 than UTC.
112 .It Dv bestring16
113 A two-byte unicode (UCS16) string in big-endian byte order.
114 .It Dv leshort
115 A two-byte value (on most systems) in little-endian byte order.
116 .It Dv lelong
117 A four-byte value (on most systems) in little-endian byte order.
118 .It Dv lequad
119 An eight-byte value (on most systems) in little-endian byte order.
120 .It Dv lefloat
121 A 32-bit (on most systems) single precision IEEE floating point number in little-endian byte order.
122 .It Dv ledouble
123 A 64-bit (on most systems) double precision IEEE floating point number in little-endian byte order.
124 .It Dv ledate
125 A four-byte value (on most systems) in little-endian byte order,
126 interpreted as a UNIX date.
127 .It Dv leqdate
128 An eight-byte value (on most systems) in little-endian byte order,
129 interpreted as a UNIX date.
130 .It Dv leldate
131 A four-byte value (on most systems) in little-endian byte order,
132 interpreted as a UNIX-style date, but interpreted as local time rather
133 than UTC.
134 .It Dv leqldate
135 An eight-byte value (on most systems) in little-endian byte order,
136 interpreted as a UNIX-style date, but interpreted as local time rather
137 than UTC.
138 .It Dv lestring16
139 A two-byte unicode (UCS16) string in little-endian byte order.
140 .It Dv melong
141 A four-byte value (on most systems) in middle-endian (PDP-11) byte order.
142 .It Dv medate
143 A four-byte value (on most systems) in middle-endian (PDP-11) byte order,
144 interpreted as a UNIX date.
145 .It Dv meldate
146 A four-byte value (on most systems) in middle-endian (PDP-11) byte order,
147 interpreted as a UNIX-style date, but interpreted as local time rather
148 than UTC.
149 .It Dv regex
150 A regular expression match in extended POSIX regular expression syntax
151 (much like egrep).
152 The type specification can be optionally followed by /[cse]*.
153 The
154 .Dq c
155 flag makes the match case insensitive, while the
156 .Dq s
157 or
158 .Dq e
159 flags update the offset to the starting or ending offsets of the
160 match (only one should be used).
161 By default, regex does not update the offset.
162 The regular expression is always tested against the first
163 .Dv N
164 lines, where
165 .Dv N
166 is the given offset, thus it
167 is only useful for (single-byte encoded) text.
168 .Dv ^
169 and
170 .Dv $
171 will match the beginning and end of individual lines, respectively,
172 not beginning and end of file.
173 .It Dv search
174 A literal string search starting at the given offset.
175 It must be followed by
176 .Dv \*[Lt]number\*[Gt]
177 which specifies how many matches shall be attempted (the range).
178 This is suitable for searching larger binary expressions with variable
179 offsets, using
180 .Dv \e
181 escapes for special characters.
182 .It Dv default
183 This is intended to be used with the text
184 .Dv x
185 (which is always true) and a message that is to be used if there are
186 no other matches.
187 .El
188 .El
189 .Pp
190 The numeric types may optionally be followed by
191 .Dv \*[Am]
192 and a numeric value,
193 to specify that the value is to be AND'ed with the
194 numeric value before any comparisons are done.
195 Prepending a
196 .Dv u
197 to the type indicates that ordered comparisons should be unsigned.
198 .Bl -tag -width ".Dv message"
199 .It Dv test
200 The value to be compared with the value from the file.
201 If the type is
202 numeric, this value
203 is specified in C form; if it is a string, it is specified as a C string
204 with the usual escapes permitted (e.g. \en for new-line).
205 .Pp
206 Numeric values
207 may be preceded by a character indicating the operation to be performed.
208 It may be
209 .Dv = ,
210 to specify that the value from the file must equal the specified value,
211 .Dv \*[Lt] ,
212 to specify that the value from the file must be less than the specified
213 value,
214 .Dv \*[Gt] ,
215 to specify that the value from the file must be greater than the specified
216 value,
217 .Dv \*[Am] ,
218 to specify that the value from the file must have set all of the bits
219 that are set in the specified value,
220 .Dv ^ ,
221 to specify that the value from the file must have clear any of the bits
222 that are set in the specified value, or
223 .Dv ~ ,
224 the value specified after is negated before tested.
225 .Dv x ,
226 to specify that any value will match.
227 If the character is omitted, it is assumed to be
228 .Dv = .
229 Operators
230 .Dv \*[Am] ,
231 .Dv ^ ,
232 and
233 .Dv ~
234 don't work with floats and doubles.
235 For all tests except
236 .Em string
237 and
238 .Em regex ,
239 operation
240 .Dv !
241 specifies that the line matches if the test does
242 .Em not
243 succeed.
244 .Pp
245 Numeric values are specified in C form; e.g.
246 .Dv 13
247 is decimal,
248 .Dv 013
249 is octal, and
250 .Dv 0x13
251 is hexadecimal.
252 .Pp
253 For string values, the byte string from the
254 file must match the specified byte string.
255 The operators
256 .Dv = ,
257 .Dv \*[Lt]
258 and
259 .Dv \*[Gt]
260 (but not
261 .Dv \*[Am] )
262 can be applied to strings.
263 The length used for matching is that of the string argument
264 in the magic file.
265 This means that a line can match any string, and
266 then presumably print that string, by doing
267 .Em \*[Gt]\e0
268 (because all strings are greater than the null string).
269 .Pp
270 The special test
271 .Em x
272 always evaluates to true.
273 .Dv message
274 The message to be printed if the comparison succeeds.
275 If the string contains a
276 .Xr printf 3
277 format specification, the value from the file (with any specified masking
278 performed) is printed using the message as the format string.
279 If the string begins with ``\\b'', the message printed is the
280 remainder of the string with no whitespace added before it: multiple
281 matches are normally separated by a single space.
282 .El
283 .Pp
284 Some file formats contain additional information which is to be printed
285 along with the file type or need additional tests to determine the true
286 file type.
287 These additional tests are introduced by one or more
288 .Em \*[Gt]
289 characters preceding the offset.
290 The number of
291 .Em \*[Gt]
292 on the line indicates the level of the test; a line with no
293 .Em \*[Gt]
294 at the beginning is considered to be at level 0.
295 Tests are arranged in a tree-like hierarchy:
296 If a the test on a line at level
297 .Em n
298 succeeds, all following tests at level
299 .Em n+1
300 are performed, and the messages printed if the tests succeed, untile a line
301 with level
302 .Em n
303 (or less) appears.
304 For more complex files, one can use empty messages to get just the
305 "if/then" effect, in the following way:
306 .Bd -literal -offset indent
307 0 string MZ
308 \*[Gt]0x18 leshort \*[Lt]0x40 MS-DOS executable
309 \*[Gt]0x18 leshort \*[Gt]0x3f extended PC executable (e.g., MS Windows)
310 .Ed
311 .Pp
312 Offsets do not need to be constant, but can also be read from the file
313 being examined.
314 If the first character following the last
315 .Em \*[Gt]
316 is a
317 .Em (
318 then the string after the parenthesis is interpreted as an indirect offset.
319 That means that the number after the parenthesis is used as an offset in
320 the file.
321 The value at that offset is read, and is used again as an offset
322 in the file.
323 Indirect offsets are of the form:
324 .Em (( x [.[bslBSL]][+\-][ y ]) .
325 The value of
326 .Em x
327 is used as an offset in the file.
328 A byte, short or long is read at that offset depending on the
329 .Em [bslBSLm]
330 type specifier.
331 The capitalized types interpret the number as a big endian
332 value, whereas the small letter versions interpret the number as a little
333 endian value;
334 the
335 .Em m
336 type interprets the number as a middle endian (PDP-11) value.
337 To that number the value of
338 .Em y
339 is added and the result is used as an offset in the file.
340 The default type if one is not specified is long.
341 .Pp
342 That way variable length structures can be examined:
343 .Bd -literal -offset indent
344 # MS Windows executables are also valid MS-DOS executables
345 0 string MZ
346 \*[Gt]0x18 leshort \*[Lt]0x40 MZ executable (MS-DOS)
347 # skip the whole block below if it is not an extended executable
348 \*[Gt]0x18 leshort \*[Gt]0x3f
349 \*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows)
350 \*[Gt]\*[Gt](0x3c.l) string LX\e0\e0 LX executable (OS/2)
351 .Ed
352 .Pp
353 This strategy of examining has one drawback: You must make sure that
354 you eventually print something, or users may get empty output (like, when
355 there is neither PE\e0\e0 nor LE\e0\e0 in the above example)
356 .Pp
357 If this indirect offset cannot be used as-is, there are simple calculations
358 possible: appending
359 .Em [+-*/%\*[Am]|^]\*[Lt]number\*[Gt]
360 inside parentheses allows one to modify
361 the value read from the file before it is used as an offset:
362 .Bd -literal -offset indent
363 # MS Windows executables are also valid MS-DOS executables
364 0 string MZ
365 # sometimes, the value at 0x18 is less that 0x40 but there's still an
366 # extended executable, simply appended to the file
367 \*[Gt]0x18 leshort \*[Lt]0x40
368 \*[Gt]\*[Gt](4.s*512) leshort 0x014c COFF executable (MS-DOS, DJGPP)
369 \*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS)
370 .Ed
371 .Pp
372 Sometimes you do not know the exact offset as this depends on the length or
373 position (when indirection was used before) of preceding fields.
374 You can specify an offset relative to the end of the last up-level
375 field using
376 .Sq \*[Am]
377 as a prefix to the offset:
378 .Bd -literal -offset indent
379 0 string MZ
380 \*[Gt]0x18 leshort \*[Gt]0x3f
381 \*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows)
382 # immediately following the PE signature is the CPU type
383 \*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x14c for Intel 80386
384 \*[Gt]\*[Gt]\*[Gt]\*[Am]0 leshort 0x184 for DEC Alpha
385 .Ed
386 .Pp
387 Indirect and relative offsets can be combined:
388 .Bd -literal -offset indent
389 0 string MZ
390 \*[Gt]0x18 leshort \*[Lt]0x40
391 \*[Gt]\*[Gt](4.s*512) leshort !0x014c MZ executable (MS-DOS)
392 # if it's not COFF, go back 512 bytes and add the offset taken
393 # from byte 2/3, which is yet another way of finding the start
394 # of the extended executable
395 \*[Gt]\*[Gt]\*[Gt]\*[Am](2.s-514) string LE LE executable (MS Windows VxD driver)
396 .Ed
397 .Pp
398 Or the other way around:
399 .Bd -literal -offset indent
400 0 string MZ
401 \*[Gt]0x18 leshort \*[Gt]0x3f
402 \*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows)
403 # at offset 0x80 (-4, since relative offsets start at the end
404 # of the up-level match) inside the LE header, we find the absolute
405 # offset to the code area, where we look for a specific signature
406 \*[Gt]\*[Gt]\*[Gt](\*[Am]0x7c.l+0x26) string UPX \eb, UPX compressed
407 .Ed
408 .Pp
409 Or even both!
410 .Bd -literal -offset indent
411 0 string MZ
412 \*[Gt]0x18 leshort \*[Gt]0x3f
413 \*[Gt]\*[Gt](0x3c.l) string LE\e0\e0 LE executable (MS-Windows)
414 # at offset 0x58 inside the LE header, we find the relative offset
415 # to a data area where we look for a specific signature
416 \*[Gt]\*[Gt]\*[Gt]\*[Am](\*[Am]0x54.l-3) string UNACE \eb, ACE self-extracting archive
417 .Ed
418 .Pp
419 Finally, if you have to deal with offset/length pairs in your file, even the
420 second value in a parenthesized expression can be taken from the file itself,
421 using another set of parentheses.
422 Note that this additional indirect offset is always relative to the
423 start of the main indirect offset.
424 .Bd -literal -offset indent
425 0 string MZ
426 \*[Gt]0x18 leshort \*[Gt]0x3f
427 \*[Gt]\*[Gt](0x3c.l) string PE\e0\e0 PE executable (MS-Windows)
428 # search for the PE section called ".idata"...
429 \*[Gt]\*[Gt]\*[Gt]\*[Am]0xf4 search/0x140 .idata
430 # ...and go to the end of it, calculated from start+length;
431 # these are located 14 and 10 bytes after the section name
432 \*[Gt]\*[Gt]\*[Gt]\*[Gt](\*[Am]0xe.l+(-4)) string PK\e3\e4 \eb, ZIP self-extracting archive
433 .Ed
434 .Sh SEE ALSO
435 .Xr file __CSECTION__
436 \- the command that reads this file.
437 .Sh BUGS
438 The formats
439 .Dv long ,
440 .Dv belong ,
441 .Dv lelong ,
442 .Dv melong ,
443 .Dv short ,
444 .Dv beshort ,
445 .Dv leshort ,
446 .Dv date ,
447 .Dv bedate ,
448 .Dv medate ,
449 .Dv ledate ,
450 .Dv beldate ,
451 .Dv leldate ,
452 and
453 .Dv meldate
454 are system-dependent; perhaps they should be specified as a number
455 of bytes (2B, 4B, etc),
456 since the files being recognized typically come from
457 a system on which the lengths are invariant.
458 .\"
459 .\" From: guy@sun.uucp (Guy Harris)
460 .\" Newsgroups: net.bugs.usg
461 .\" Subject: /etc/magic's format isn't well documented
462 .\" Message-ID: <2752@sun.uucp>
463 .\" Date: 3 Sep 85 08:19:07 GMT
464 .\" Organization: Sun Microsystems, Inc.
465 .\" Lines: 136
466 .\"
467 .\" Here's a manual page for the format accepted by the "file" made by adding
468 .\" the changes I posted to the S5R2 version.
469 .\"
470 .\" Modified for Ian Darwin's version of the file command.
471 .\" @(#)$Id: magic.man,v 1.39 2007/11/08 00:31:37 christos Exp $