1 .\" Copyright (c) 2003-2007 Tim Kientzle
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
31 .Nm libarchive_internals
32 .Nd description of libarchive internal interfaces
36 library provides a flexible interface for reading and writing
37 streaming archive files such as tar and cpio.
38 Internally, it follows a modular layered design that should
39 make it easy to add new archive and compression formats.
40 .Sh GENERAL ARCHITECTURE
41 Externally, libarchive exposes most operations through an
42 opaque, object-style interface.
45 objects store information about a single filesystem object.
46 The rest of the library provides facilities to write
48 objects to archive files,
49 read them from archive files,
50 and write them to disk.
51 (There are plans to add a facility to read
53 objects from disk as well.)
55 The read and write APIs each have four layers: a public API
56 layer, a format layer that understands the archive file format,
57 a compression layer, and an I/O layer.
58 The I/O layer is completely exposed to clients who can replace
59 it entirely with their own functions.
61 In order to provide as much consistency as possible for clients,
62 some public functions are virtualized.
63 Eventually, it should be possible for clients to open
64 an archive or disk writer, and then use a single set of
65 code to select and write entries, regardless of the target.
67 From the outside, clients use the
71 object to read entries and bodies from an archive stream.
76 object, which holds all read-specific data.
77 The API has four layers:
78 The lowest layer is the I/O layer.
79 This layer can be overridden by clients, but most clients use
80 the packaged I/O callbacks provided, for example, by
81 .Xr archive_read_open_memory 3 ,
83 .Xr archive_read_open_fd 3 .
84 The compression layer calls the I/O layer to
85 read bytes and decompresses them for the format layer.
86 The format layer unpacks a stream of uncompressed bytes and
89 objects from the incoming data.
90 The API layer tracks overall state
91 (for example, it prevents clients from reading data before reading a header)
92 and invokes the format and compression layer operations
93 through registered function pointers.
94 In particular, the API layer drives the format-detection process:
95 When opening the archive, it reads an initial block of data
96 and offers it to each registered compression handler.
97 The one with the highest bid is initialized with the first block.
98 Similarly, the format handlers are polled to see which handler
99 is the best for each header request.
100 (Note that a single file can have entries handled by different
102 this allows a simple handler for a generic version of a format
103 with more complex handlers implemented independently for
104 extended sub-formats.)
105 .Ss I/O Layer and Client Callbacks
106 The read API goes to some lengths to be nice to clients.
107 As a result, there are few restrictions on the behavior of
108 the client callbacks.
110 The client read callback is expected to provide a block
111 of data on each call.
112 A zero-length return does indicate end of file, but otherwise
113 blocks may be as small as one byte or as large as the entire file.
114 In particular, blocks may be of different sizes.
116 The client skip callback returns the number of bytes actually
117 skipped, which may be much smaller than the skip requested.
118 The only requirement is that the skip not be larger.
119 The skip callback must never be invoked with a negative value.
121 Keep in mind that not all clients are reading from disk:
122 clients reading from networks may provide different-sized
123 blocks on every request and cannot skip at all;
124 advanced clients may use
126 to read the entire file into memory at once and return the
127 entire file to libarchive as a single block;
128 other clients may begin asynchronous I/O operations for the
129 next block on each request.
130 .Ss Decompresssion Layer
131 The decompression layer not only handles decompression,
132 it also buffers data so that the format handlers see a
133 much nicer I/O model.
134 The decompression API is a two stage peek/consume model.
135 A read_ahead request specifies a minimum read amount;
136 the decompression layer must provide a pointer to at least
138 If more data is immediately available, it should return more:
139 the format layer handles bulk data reads by asking for a minimum
140 of one byte and then copying as much data as is available.
142 A subsequent call to the
144 function advances the read pointer.
145 Note that data returned from a
147 call is guaranteed to remain in place until
152 should not cause the data to move.
154 Skip requests must always be handled exactly.
155 Decompression handlers that cannot seek forward should
156 not register a skip handler;
157 the API layer fills in a generic skip handler that reads and discards data.
159 A decompression handler has a specific lifecycle:
160 .Bl -tag -compact -width indent
161 .It Registration/Configuration
162 When the client invokes the public support function,
163 the decompression handler invokes the internal
164 .Fn __archive_read_register_compression
165 function to provide bid and initialization functions.
166 This function returns
168 on error or else a pointer to a
169 .Cm struct decompressor_t .
170 This structure contains a
172 slot that can be used for storing any customization information.
174 The bid function is invoked with a pointer and size of a block of data.
175 The decompressor can access its config data
181 The bid function is otherwise stateless.
182 In particular, it must not perform any I/O operations.
184 The value returned by the bid function indicates its suitability
185 for handling this data stream.
186 A bid of zero will ensure that this decompressor is never invoked.
187 Return zero if magic number checks fail.
188 Otherwise, your initial implementation should return the number of bits
190 For example, if you verify two full bytes and three bits of another
192 Note that the initial block may be very short;
193 be careful to only inspect the data you are given.
194 (The current decompressors require two bytes for correct bidding.)
196 The winning bidder will have its init function called.
197 This function should initialize the remaining slots of the
198 .Va struct decompressor_t
199 object pointed to by the
204 In particular, it should allocate any working data it needs
207 slot of that structure.
208 The init function is called with the block of data that
209 was used for tasting.
210 At this point, the decompressor is responsible for all I/O
211 requests to the client callbacks.
212 The decompressor is free to read more data as and when
214 .It Satisfy I/O requests
215 The format handler will invoke the
222 The finish method is called only once when the archive is closed.
223 It should release anything stored in the
230 It should not invoke the client close callback.
233 The read formats have a similar lifecycle to the decompression handlers:
234 .Bl -tag -compact -width indent
236 Allocate your private data and initialize your pointers.
238 Formats bid by invoking the
240 decompression method but not calling the
243 This allows each bidder to look ahead in the input stream.
244 Bidders should not look further ahead than necessary, as long
245 look aheads put pressure on the compression layer to buffer
247 Most formats only require a few hundred bytes of look ahead;
248 look aheads of a few kilobytes are reasonable.
249 (The ISO9660 reader sometimes looks ahead by 48k, which
250 should be considered an upper limit.)
251 Note that the bidder is invoked for every entry.
252 For many formats, this is inappropriate; if you can only bid at
253 the beginning of the file, store your bid value and check that
254 each time your bid function is called.
255 For example, the ISO9660 reader initializes a
257 value to -1 at registration time;
258 each time the bid function is called, the bid value is returned
259 immediately if it is zero or larger.
261 The header read is usually the most complex part of any format.
262 There are a few strategies worth mentioning:
263 For formats such as tar or cpio, reading and parsing the header is
264 straightforward since headers alternate with data.
265 For formats that store all header data at the beginning of the file,
266 the first header read request may have to read all headers into
267 memory and store that data, sorted by the location of the file
269 Subsequent header read requests will skip forward to the
270 beginning of the file data and return the corresponding header.
272 The read data interface supports sparse files; this requires that
273 each call return a block of data specifying the file offset and
275 This may require you to carefully track the location so that you
276 can return accurate file offsets for each read.
277 Remember that the decompressor will return as much data as it has.
278 Generally, you will want to request one byte,
279 examine the return value to see how much data is available, and
280 possibly trim that to the amount you can use.
281 You should invoke consume for each block just before you return it.
283 The skip data call should skip over all file data and trailing padding.
284 This is called automatically by the API layer just before each
286 It is also called in response to the client calling the public
290 On cleanup, the format should release all of its allocated memory.
294 .Sh WRITE ARCHITECTURE
295 The write API has a similar set of four layers:
296 an API layer, a format layer, a compression layer, and an I/O layer.
297 The registration here is much simpler because only
298 one format and one compression can be registered at a time.
299 .Ss I/O Layer and Client Callbacks
300 XXX To be written XXX
301 .Ss Compression Layer
302 XXX To be written XXX
304 XXX To be written XXX
306 XXX To be written XXX
307 .Sh WRITE_DISK ARCHITECTURE
308 The write_disk API is intended to look just like the write API
310 Since it does not handle multiple formats or compression, it
311 is not layered internally.
317 .Nm archive_write_disk
318 objects all contain an initial
320 object which provides common support for a set of standard services.
321 (Recall that ANSI/ISO C90 guarantees that you can cast freely between
322 a pointer to a structure and a pointer to the first element of that
326 object has a magic value that indicates which API this object
328 slots for storing error information,
329 and function pointers for virtualized API functions.
330 .Sh MISCELLANEOUS NOTES
331 Connecting existing archiving libraries into libarchive is generally
333 In particular, many existing libraries strongly assume that you
334 are reading from a file; they seek forwards and backwards as necessary
335 to locate various pieces of information.
336 In contrast, libarchive never seeks backwards in its input, which
337 sometimes requires very different approaches.
339 For example, libarchive's ISO9660 support operates very differently
340 from most ISO9660 readers.
341 The libarchive support utilizes a work-queue design that
342 keeps a list of known entries sorted by their location in the input.
343 Whenever libarchive's ISO9660 implementation is asked for the next
344 header, checks this list to find the next item on the disk.
345 Directories are parsed when they are encountered and new
346 items are added to the list.
347 This design relies heavily on the ISO9660 image being optimized so that
348 directories always occur earlier on the disk than the files they
351 Depending on the specific format, such approaches may not be possible.
352 The ZIP format specification, for example, allows archivers to store
353 key information only at the end of the file.
354 In theory, it is possible to create ZIP archives that cannot
355 be read without seeking.
356 Fortunately, such archives are very rare, and libarchive can read
357 most ZIP archives, though it cannot always extract as much information
358 as a dedicated ZIP program.
361 .Xr archive_entry 3 ,
363 .Xr archive_write 3 ,
364 .Xr archive_write_disk 3
368 library first appeared in
374 library was written by
375 .An Tim Kientzle Aq kientzle@acm.org .