| blob | dc17e30bd5fed34bb5ff174b7ba727904164153f |
1 ext4: import journal chapter from wiki page
3 From: Darrick J. Wong <darrick.wong@oracle.com>
5 Import the chapter about the journal from the on-disk format wiki
6 page into the kernel documentation.
8 Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
9 Signed-off-by: Theodore Ts'o <tytso@mit.edu>
10 ---
11 Documentation/filesystems/ext4/ondisk/globals.rst | 1
12 Documentation/filesystems/ext4/ondisk/journal.rst | 611 +++++++++++++++++++++
13 2 files changed, 612 insertions(+)
14 create mode 100644 Documentation/filesystems/ext4/ondisk/journal.rst
17 diff --git a/Documentation/filesystems/ext4/ondisk/globals.rst b/Documentation/filesystems/ext4/ondisk/globals.rst
18 index fe6e107bf515..368bf7662b96 100644
19 --- a/Documentation/filesystems/ext4/ondisk/globals.rst
20 +++ b/Documentation/filesystems/ext4/ondisk/globals.rst
21 @@ -10,3 +10,4 @@ have static metadata at fixed locations.
22 .. include:: group_descr.rst
23 .. include:: bitmaps.rst
24 .. include:: mmp.rst
25 +.. include:: journal.rst
26 diff --git a/Documentation/filesystems/ext4/ondisk/journal.rst b/Documentation/filesystems/ext4/ondisk/journal.rst
27 new file mode 100644
28 index 000000000000..e7031af86876
29 --- /dev/null
30 +++ b/Documentation/filesystems/ext4/ondisk/journal.rst
31 @@ -0,0 +1,611 @@
32 +.. SPDX-License-Identifier: GPL-2.0
33 +
34 +Journal (jbd2)
35 +--------------
36 +
37 +Introduced in ext3, the ext4 filesystem employs a journal to protect the
38 +filesystem against corruption in the case of a system crash. A small
39 +continuous region of disk (default 128MiB) is reserved inside the
40 +filesystem as a place to land “important” data writes on-disk as quickly
41 +as possible. Once the important data transaction is fully written to the
42 +disk and flushed from the disk write cache, a record of the data being
43 +committed is also written to the journal. At some later point in time,
44 +the journal code writes the transactions to their final locations on
45 +disk (this could involve a lot of seeking or a lot of small
46 +read-write-erases) before erasing the commit record. Should the system
47 +crash during the second slow write, the journal can be replayed all the
48 +way to the latest commit record, guaranteeing the atomicity of whatever
49 +gets written through the journal to the disk. The effect of this is to
50 +guarantee that the filesystem does not become stuck midway through a
51 +metadata update.
52 +
53 +For performance reasons, ext4 by default only writes filesystem metadata
54 +through the journal. This means that file data blocks are /not/
55 +guaranteed to be in any consistent state after a crash. If this default
56 +guarantee level (``data=ordered``) is not satisfactory, there is a mount
57 +option to control journal behavior. If ``data=journal``, all data and
58 +metadata are written to disk through the journal. This is slower but
59 +safest. If ``data=writeback``, dirty data blocks are not flushed to the
60 +disk before the metadata are written to disk through the journal.
61 +
62 +The journal inode is typically inode 8. The first 68 bytes of the
63 +journal inode are replicated in the ext4 superblock. The journal itself
64 +is normal (but hidden) file within the filesystem. The file usually
65 +consumes an entire block group, though mke2fs tries to put it in the
66 +middle of the disk.
67 +
68 +All fields in jbd2 are written to disk in big-endian order. This is the
69 +opposite of ext4.
70 +
71 +NOTE: Both ext4 and ocfs2 use jbd2.
72 +
73 +The maximum size of a journal embedded in an ext4 filesystem is 2^32
74 +blocks. jbd2 itself does not seem to care.
75 +
76 +Layout
77 +~~~~~~
78 +
79 +Generally speaking, the journal has this format:
80 +
81 +.. list-table::
82 + :widths: 1 1 78
83 + :header-rows: 1
84 +
85 + * - Superblock
86 + - descriptor\_block (data\_blocks or revocation\_block) [more data or
87 + revocations] commmit\_block
88 + - [more transactions...]
89 + * -
90 + - One transaction
91 + -
92 +
93 +Notice that a transaction begins with either a descriptor and some data,
94 +or a block revocation list. A finished transaction always ends with a
95 +commit. If there is no commit record (or the checksums don't match), the
96 +transaction will be discarded during replay.
97 +
98 +External Journal
99 +~~~~~~~~~~~~~~~~
100 +
101 +Optionally, an ext4 filesystem can be created with an external journal
102 +device (as opposed to an internal journal, which uses a reserved inode).
103 +In this case, on the filesystem device, ``s_journal_inum`` should be
104 +zero and ``s_journal_uuid`` should be set. On the journal device there
105 +will be an ext4 super block in the usual place, with a matching UUID.
106 +The journal superblock will be in the next full block after the
107 +superblock.
108 +
109 +.. list-table::
110 + :widths: 1 1 1 1 76
111 + :header-rows: 1
112 +
113 + * - 1024 bytes of padding
114 + - ext4 Superblock
115 + - Journal Superblock
116 + - descriptor\_block (data\_blocks or revocation\_block) [more data or
117 + revocations] commmit\_block
118 + - [more transactions...]
119 + * -
120 + -
121 + -
122 + - One transaction
123 + -
124 +
125 +Block Header
126 +~~~~~~~~~~~~
127 +
128 +Every block in the journal starts with a common 12-byte header
129 +``struct journal_header_s``:
130 +
131 +.. list-table::
132 + :widths: 1 1 1 77
133 + :header-rows: 1
134 +
135 + * - Offset
136 + - Type
137 + - Name
138 + - Description
139 + * - 0x0
140 + - \_\_be32
141 + - h\_magic
142 + - jbd2 magic number, 0xC03B3998.
143 + * - 0x4
144 + - \_\_be32
145 + - h\_blocktype
146 + - Description of what this block contains. See the jbd2_blocktype_ table
147 + below.
148 + * - 0x8
149 + - \_\_be32
150 + - h\_sequence
151 + - The transaction ID that goes with this block.
152 +
153 +.. _jbd2_blocktype:
154 +
155 +The journal block type can be any one of:
156 +
157 +.. list-table::
158 + :widths: 1 79
159 + :header-rows: 1
160 +
161 + * - Value
162 + - Description
163 + * - 1
164 + - Descriptor. This block precedes a series of data blocks that were
165 + written through the journal during a transaction.
166 + * - 2
167 + - Block commit record. This block signifies the completion of a
168 + transaction.
169 + * - 3
170 + - Journal superblock, v1.
171 + * - 4
172 + - Journal superblock, v2.
173 + * - 5
174 + - Block revocation records. This speeds up recovery by enabling the
175 + journal to skip writing blocks that were subsequently rewritten.
176 +
177 +Super Block
178 +~~~~~~~~~~~
179 +
180 +The super block for the journal is much simpler as compared to ext4's.
181 +The key data kept within are size of the journal, and where to find the
182 +start of the log of transactions.
183 +
184 +The journal superblock is recorded as ``struct journal_superblock_s``,
185 +which is 1024 bytes long:
186 +
187 +.. list-table::
188 + :widths: 1 1 1 77
189 + :header-rows: 1
190 +
191 + * - Offset
192 + - Type
193 + - Name
194 + - Description
195 + * -
196 + -
197 + -
198 + - Static information describing the journal.
199 + * - 0x0
200 + - journal\_header\_t (12 bytes)
201 + - s\_header
202 + - Common header identifying this as a superblock.
203 + * - 0xC
204 + - \_\_be32
205 + - s\_blocksize
206 + - Journal device block size.
207 + * - 0x10
208 + - \_\_be32
209 + - s\_maxlen
210 + - Total number of blocks in this journal.
211 + * - 0x14
212 + - \_\_be32
213 + - s\_first
214 + - First block of log information.
215 + * -
216 + -
217 + -
218 + - Dynamic information describing the current state of the log.
219 + * - 0x18
220 + - \_\_be32
221 + - s\_sequence
222 + - First commit ID expected in log.
223 + * - 0x1C
224 + - \_\_be32
225 + - s\_start
226 + - Block number of the start of log. Contrary to the comments, this field
227 + being zero does not imply that the journal is clean!
228 + * - 0x20
229 + - \_\_be32
230 + - s\_errno
231 + - Error value, as set by jbd2\_journal\_abort().
232 + * -
233 + -
234 + -
235 + - The remaining fields are only valid in a v2 superblock.
236 + * - 0x24
237 + - \_\_be32
238 + - s\_feature\_compat;
239 + - Compatible feature set. See the table jbd2_compat_ below.
240 + * - 0x28
241 + - \_\_be32
242 + - s\_feature\_incompat
243 + - Incompatible feature set. See the table jbd2_incompat_ below.
244 + * - 0x2C
245 + - \_\_be32
246 + - s\_feature\_ro\_compat
247 + - Read-only compatible feature set. There aren't any of these currently.
248 + * - 0x30
249 + - \_\_u8
250 + - s\_uuid[16]
251 + - 128-bit uuid for journal. This is compared against the copy in the ext4
252 + super block at mount time.
253 + * - 0x40
254 + - \_\_be32
255 + - s\_nr\_users
256 + - Number of file systems sharing this journal.
257 + * - 0x44
258 + - \_\_be32
259 + - s\_dynsuper
260 + - Location of dynamic super block copy. (Not used?)
261 + * - 0x48
262 + - \_\_be32
263 + - s\_max\_transaction
264 + - Limit of journal blocks per transaction. (Not used?)
265 + * - 0x4C
266 + - \_\_be32
267 + - s\_max\_trans\_data
268 + - Limit of data blocks per transaction. (Not used?)
269 + * - 0x50
270 + - \_\_u8
271 + - s\_checksum\_type
272 + - Checksum algorithm used for the journal. See jbd2_checksum_type_ for
273 + more info.
274 + * - 0x51
275 + - \_\_u8[3]
276 + - s\_padding2
277 + -
278 + * - 0x54
279 + - \_\_u32
280 + - s\_padding[42]
281 + -
282 + * - 0xFC
283 + - \_\_be32
284 + - s\_checksum
285 + - Checksum of the entire superblock, with this field set to zero.
286 + * - 0x100
287 + - \_\_u8
288 + - s\_users[16\*48]
289 + - ids of all file systems sharing the log. e2fsprogs/Linux don't allow
290 + shared external journals, but I imagine Lustre (or ocfs2?), which use
291 + the jbd2 code, might.
292 +
293 +.. _jbd2_compat:
294 +
295 +The journal compat features are any combination of the following:
296 +
297 +.. list-table::
298 + :widths: 1 79
299 + :header-rows: 1
300 +
301 + * - Value
302 + - Description
303 + * - 0x1
304 + - Journal maintains checksums on the data blocks.
305 + (JBD2\_FEATURE\_COMPAT\_CHECKSUM)
306 +
307 +.. _jbd2_incompat:
308 +
309 +The journal incompat features are any combination of the following:
310 +
311 +.. list-table::
312 + :widths: 1 79
313 + :header-rows: 1
314 +
315 + * - Value
316 + - Description
317 + * - 0x1
318 + - Journal has block revocation records. (JBD2\_FEATURE\_INCOMPAT\_REVOKE)
319 + * - 0x2
320 + - Journal can deal with 64-bit block numbers.
321 + (JBD2\_FEATURE\_INCOMPAT\_64BIT)
322 + * - 0x4
323 + - Journal commits asynchronously. (JBD2\_FEATURE\_INCOMPAT\_ASYNC\_COMMIT)
324 + * - 0x8
325 + - This journal uses v2 of the checksum on-disk format. Each journal
326 + metadata block gets its own checksum, and the block tags in the
327 + descriptor table contain checksums for each of the data blocks in the
328 + journal. (JBD2\_FEATURE\_INCOMPAT\_CSUM\_V2)
329 + * - 0x10
330 + - This journal uses v3 of the checksum on-disk format. This is the same as
331 + v2, but the journal block tag size is fixed regardless of the size of
332 + block numbers. (JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3)
333 +
334 +.. _jbd2_checksum_type:
335 +
336 +Journal checksum type codes are one of the following. crc32 or crc32c are the
337 +most likely choices.
338 +
339 +.. list-table::
340 + :widths: 1 79
341 + :header-rows: 1
342 +
343 + * - Value
344 + - Description
345 + * - 1
346 + - CRC32
347 + * - 2
348 + - MD5
349 + * - 3
350 + - SHA1
351 + * - 4
352 + - CRC32C
353 +
354 +Descriptor Block
355 +~~~~~~~~~~~~~~~~
356 +
357 +The descriptor block contains an array of journal block tags that
358 +describe the final locations of the data blocks that follow in the
359 +journal. Descriptor blocks are open-coded instead of being completely
360 +described by a data structure, but here is the block structure anyway.
361 +Descriptor blocks consume at least 36 bytes, but use a full block:
362 +
363 +.. list-table::
364 + :widths: 1 1 1 77
365 + :header-rows: 1
366 +
367 + * - Offset
368 + - Type
369 + - Name
370 + - Descriptor
371 + * - 0x0
372 + - journal\_header\_t
373 + - (open coded)
374 + - Common block header.
375 + * - 0xC
376 + - struct journal\_block\_tag\_s
377 + - open coded array[]
378 + - Enough tags either to fill up the block or to describe all the data
379 + blocks that follow this descriptor block.
380 +
381 +Journal block tags have any of the following formats, depending on which
382 +journal feature and block tag flags are set.
383 +
384 +If JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3 is set, the journal block tag is
385 +defined as ``struct journal_block_tag3_s``, which looks like the
386 +following. The size is 16 or 32 bytes.
387 +
388 +.. list-table::
389 + :widths: 1 1 1 77
390 + :header-rows: 1
391 +
392 + * - Offset
393 + - Type
394 + - Name
395 + - Descriptor
396 + * - 0x0
397 + - \_\_be32
398 + - t\_blocknr
399 + - Lower 32-bits of the location of where the corresponding data block
400 + should end up on disk.
401 + * - 0x4
402 + - \_\_be32
403 + - t\_flags
404 + - Flags that go with the descriptor. See the table jbd2_tag_flags_ for
405 + more info.
406 + * - 0x8
407 + - \_\_be32
408 + - t\_blocknr\_high
409 + - Upper 32-bits of the location of where the corresponding data block
410 + should end up on disk. This is zero if JBD2\_FEATURE\_INCOMPAT\_64BIT is
411 + not enabled.
412 + * - 0xC
413 + - \_\_be32
414 + - t\_checksum
415 + - Checksum of the journal UUID, the sequence number, and the data block.
416 + * -
417 + -
418 + -
419 + - This field appears to be open coded. It always comes at the end of the
420 + tag, after t_checksum. This field is not present if the "same UUID" flag
421 + is set.
422 + * - 0x8 or 0xC
423 + - char
424 + - uuid[16]
425 + - A UUID to go with this tag. This field appears to be copied from the
426 + ``j_uuid`` field in ``struct journal_s``, but only tune2fs touches that
427 + field.
428 +
429 +.. _jbd2_tag_flags:
430 +
431 +The journal tag flags are any combination of the following:
432 +
433 +.. list-table::
434 + :widths: 1 79
435 + :header-rows: 1
436 +
437 + * - Value
438 + - Description
439 + * - 0x1
440 + - On-disk block is escaped. The first four bytes of the data block just
441 + happened to match the jbd2 magic number.
442 + * - 0x2
443 + - This block has the same UUID as previous, therefore the UUID field is
444 + omitted.
445 + * - 0x4
446 + - The data block was deleted by the transaction. (Not used?)
447 + * - 0x8
448 + - This is the last tag in this descriptor block.
449 +
450 +If JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3 is NOT set, the journal block tag
451 +is defined as ``struct journal_block_tag_s``, which looks like the
452 +following. The size is 8, 12, 24, or 28 bytes:
453 +
454 +.. list-table::
455 + :widths: 1 1 1 77
456 + :header-rows: 1
457 +
458 + * - Offset
459 + - Type
460 + - Name
461 + - Descriptor
462 + * - 0x0
463 + - \_\_be32
464 + - t\_blocknr
465 + - Lower 32-bits of the location of where the corresponding data block
466 + should end up on disk.
467 + * - 0x4
468 + - \_\_be16
469 + - t\_checksum
470 + - Checksum of the journal UUID, the sequence number, and the data block.
471 + Note that only the lower 16 bits are stored.
472 + * - 0x6
473 + - \_\_be16
474 + - t\_flags
475 + - Flags that go with the descriptor. See the table jbd2_tag_flags_ for
476 + more info.
477 + * -
478 + -
479 + -
480 + - This next field is only present if the super block indicates support for
481 + 64-bit block numbers.
482 + * - 0x8
483 + - \_\_be32
484 + - t\_blocknr\_high
485 + - Upper 32-bits of the location of where the corresponding data block
486 + should end up on disk.
487 + * -
488 + -
489 + -
490 + - This field appears to be open coded. It always comes at the end of the
491 + tag, after t_flags or t_blocknr_high. This field is not present if the
492 + "same UUID" flag is set.
493 + * - 0x8 or 0xC
494 + - char
495 + - uuid[16]
496 + - A UUID to go with this tag. This field appears to be copied from the
497 + ``j_uuid`` field in ``struct journal_s``, but only tune2fs touches that
498 + field.
499 +
500 +If JBD2\_FEATURE\_INCOMPAT\_CSUM\_V2 or
501 +JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3 are set, the end of the block is a
502 +``struct jbd2_journal_block_tail``, which looks like this:
503 +
504 +.. list-table::
505 + :widths: 1 1 1 77
506 + :header-rows: 1
507 +
508 + * - Offset
509 + - Type
510 + - Name
511 + - Descriptor
512 + * - 0x0
513 + - \_\_be32
514 + - t\_checksum
515 + - Checksum of the journal UUID + the descriptor block, with this field set
516 + to zero.
517 +
518 +Data Block
519 +~~~~~~~~~~
520 +
521 +In general, the data blocks being written to disk through the journal
522 +are written verbatim into the journal file after the descriptor block.
523 +However, if the first four bytes of the block match the jbd2 magic
524 +number then those four bytes are replaced with zeroes and the “escaped”
525 +flag is set in the descriptor block tag.
526 +
527 +Revocation Block
528 +~~~~~~~~~~~~~~~~
529 +
530 +A revocation block is used to prevent replay of a block in an earlier
531 +transaction. This is used to mark blocks that were journalled at one
532 +time but are no longer journalled. Typically this happens if a metadata
533 +block is freed and re-allocated as a file data block; in this case, a
534 +journal replay after the file block was written to disk will cause
535 +corruption.
536 +
537 +**NOTE**: This mechanism is NOT used to express “this journal block is
538 +superseded by this other journal block”, as the author (djwong)
539 +mistakenly thought. Any block being added to a transaction will cause
540 +the removal of all existing revocation records for that block.
541 +
542 +Revocation blocks are described in
543 +``struct jbd2_journal_revoke_header_s``, are at least 16 bytes in
544 +length, but use a full block:
545 +
546 +.. list-table::
547 + :widths: 1 1 1 77
548 + :header-rows: 1
549 +
550 + * - Offset
551 + - Type
552 + - Name
553 + - Description
554 + * - 0x0
555 + - journal\_header\_t
556 + - r\_header
557 + - Common block header.
558 + * - 0xC
559 + - \_\_be32
560 + - r\_count
561 + - Number of bytes used in this block.
562 + * - 0x10
563 + - \_\_be32 or \_\_be64
564 + - blocks[0]
565 + - Blocks to revoke.
566 +
567 +After r\_count is a linear array of block numbers that are effectively
568 +revoked by this transaction. The size of each block number is 8 bytes if
569 +the superblock advertises 64-bit block number support, or 4 bytes
570 +otherwise.
571 +
572 +If JBD2\_FEATURE\_INCOMPAT\_CSUM\_V2 or
573 +JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3 are set, the end of the revocation
574 +block is a ``struct jbd2_journal_revoke_tail``, which has this format:
575 +
576 +.. list-table::
577 + :widths: 1 1 1 77
578 + :header-rows: 1
579 +
580 + * - Offset
581 + - Type
582 + - Name
583 + - Description
584 + * - 0x0
585 + - \_\_be32
586 + - r\_checksum
587 + - Checksum of the journal UUID + revocation block
588 +
589 +Commit Block
590 +~~~~~~~~~~~~
591 +
592 +The commit block is a sentry that indicates that a transaction has been
593 +completely written to the journal. Once this commit block reaches the
594 +journal, the data stored with this transaction can be written to their
595 +final locations on disk.
596 +
597 +The commit block is described by ``struct commit_header``, which is 32
598 +bytes long (but uses a full block):
599 +
600 +.. list-table::
601 + :widths: 1 1 1 77
602 + :header-rows: 1
603 +
604 + * - Offset
605 + - Type
606 + - Name
607 + - Descriptor
608 + * - 0x0
609 + - journal\_header\_s
610 + - (open coded)
611 + - Common block header.
612 + * - 0xC
613 + - unsigned char
614 + - h\_chksum\_type
615 + - The type of checksum to use to verify the integrity of the data blocks
616 + in the transaction. See jbd2_checksum_type_ for more info.
617 + * - 0xD
618 + - unsigned char
619 + - h\_chksum\_size
620 + - The number of bytes used by the checksum. Most likely 4.
621 + * - 0xE
622 + - unsigned char
623 + - h\_padding[2]
624 + -
625 + * - 0x10
626 + - \_\_be32
627 + - h\_chksum[JBD2\_CHECKSUM\_BYTES]
628 + - 32 bytes of space to store checksums. If
629 + JBD2\_FEATURE\_INCOMPAT\_CSUM\_V2 or JBD2\_FEATURE\_INCOMPAT\_CSUM\_V3
630 + are set, the first ``__be32`` is the checksum of the journal UUID and
631 + the entire commit block, with this field zeroed. If
632 + JBD2\_FEATURE\_COMPAT\_CHECKSUM is set, the first ``__be32`` is the
633 + crc32 of all the blocks already written to the transaction.
634 + * - 0x30
635 + - \_\_be64
636 + - h\_commit\_sec
637 + - The time that the transaction was committed, in seconds since the epoch.
638 + * - 0x38
639 + - \_\_be32
640 + - h\_commit\_nsec
641 + - Nanoseconds component of the above timestamp.
642 +