| blob | c20bd37a7971e68803d67932217949e30de232ee |
1 ..
2 Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc.
3 All rights reserved.
5 This file is licensed via The FreeBSD Documentation License, the full
6 text of which is included at the end of this document.
8 ====================================
9 Dirty Bitmaps and Incremental Backup
10 ====================================
12 Dirty Bitmaps are in-memory objects that track writes to block devices. They
13 can be used in conjunction with various block job operations to perform
14 incremental or differential backup regimens.
16 This document explains the conceptual mechanisms, as well as up-to-date,
17 complete and comprehensive documentation on the API to manipulate them.
18 (Hopefully, the "why", "what", and "how".)
20 The intended audience for this document is developers who are adding QEMU
21 backup features to management applications, or power users who run and
22 administer QEMU directly via QMP.
24 .. contents::
26 Overview
27 --------
29 Bitmaps are bit vectors where each '1' bit in the vector indicates a modified
30 ("dirty") segment of the corresponding block device. The size of the segment
31 that is tracked is the granularity of the bitmap. If the granularity of a
32 bitmap is 64K, each '1' bit means that a 64K region as a whole may have
33 changed in some way, possibly by as little as one byte.
35 Smaller granularities mean more accurate tracking of modified disk data, but
36 requires more computational overhead and larger bitmap sizes. Larger
37 granularities mean smaller bitmap sizes, but less targeted backups.
39 The size of a bitmap (in bytes) can be computed as such:
40 ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8)
42 e.g. the size of a 64KiB granularity bitmap on a 2TiB image is:
43 ``size`` = ((2147483648K / 64K) / 8)
44 = 4194304B = 4MiB.
46 QEMU uses these bitmaps when making incremental backups to know which sections
47 of the file to copy out. They are not enabled by default and must be
48 explicitly added in order to begin tracking writes.
50 Bitmaps can be created at any time and can be attached to any arbitrary block
51 node in the storage graph, but are most useful conceptually when attached to
52 the root node attached to the guest's storage device model.
54 That is to say: It's likely most useful to track the guest's writes to disk,
55 but you could theoretically track things like qcow2 metadata changes by
56 attaching the bitmap elsewhere in the storage graph. This is beyond the scope
57 of this document.
59 QEMU supports persisting these bitmaps to disk via the qcow2 image format.
60 Bitmaps which are stored or loaded in this way are called "persistent",
61 whereas bitmaps that are not are called "transient".
63 QEMU also supports the migration of both transient bitmaps (tracking any
64 arbitrary image format) or persistent bitmaps (qcow2) via live migration.
66 Supported Image Formats
67 -----------------------
69 QEMU supports all documented features below on the qcow2 image format.
71 However, qcow2 is only strictly necessary for the persistence feature, which
72 writes bitmap data to disk upon close. If persistence is not required for a
73 specific use case, all bitmap features excepting persistence are available for
74 any arbitrary image format.
76 For example, Dirty Bitmaps can be combined with the 'raw' image format, but
77 any changes to the bitmap will be discarded upon exit.
79 .. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent
80 bitmaps are available only on qcow2 images.
82 Dirty Bitmap Names
83 ------------------
85 Bitmap objects need a method to reference them in the API. All API-created and
86 managed bitmaps have a human-readable name chosen by the user at creation
87 time.
89 - A bitmap's name is unique to the node, but bitmaps attached to different
90 nodes can share the same name. Therefore, all bitmaps are addressed via
91 their (node, name) pair.
93 - The name of a user-created bitmap cannot be empty ("").
95 - Transient bitmaps can have JSON unicode names that are effectively not
96 length limited. (QMP protocol may restrict messages to less than 64MiB.)
98 - Persistent storage formats may impose their own requirements on bitmap names
99 and namespaces. Presently, only qcow2 supports persistent bitmaps. See
100 docs/interop/qcow2.txt for more details on restrictions. Notably:
102 - qcow2 bitmap names are limited to between 1 and 1023 bytes long.
104 - No two bitmaps saved to the same qcow2 file may share the same name.
106 - QEMU occasionally uses bitmaps for internal use which have no name. They are
107 hidden from API query calls, cannot be manipulated by the external API, are
108 never persistent, nor ever migrated.
110 Bitmap Status
111 -------------
113 Dirty Bitmap objects can be queried with the QMP command `query-block
114 <qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the
115 `BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure.
117 This struct shows the name, granularity, and dirty byte count for each bitmap.
118 Additionally, it shows several boolean status indicators:
120 - ``recording``: This bitmap is recording writes.
121 - ``busy``: This bitmap is in-use by an operation.
122 - ``persistent``: This bitmap is a persistent type.
123 - ``inconsistent``: This bitmap is corrupted and cannot be used.
125 The ``+busy`` status prohibits you from deleting, clearing, or otherwise
126 modifying a bitmap, and happens when the bitmap is being used for a backup
127 operation or is in the process of being loaded from a migration. Many of the
128 commands documented below will refuse to work on such bitmaps.
130 The ``+inconsistent`` status similarly prohibits almost all operations,
131 notably allowing only the ``block-dirty-bitmap-remove`` operation.
133 There is also a deprecated ``status`` field of type `DirtyBitmapStatus
134 <qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had
135 five visible states:
137 #. ``Frozen``: This bitmap is currently in-use by an operation and is
138 immutable. It can't be deleted, renamed, reset, etc.
140 (This is now ``+busy``.)
142 #. ``Disabled``: This bitmap is not recording new writes.
144 (This is now ``-recording -busy``.)
146 #. ``Active``: This bitmap is recording new writes.
148 (This is now ``+recording -busy``.)
150 #. ``Locked``: This bitmap is in-use by an operation, and is immutable.
151 The difference from "Frozen" was primarily implementation details.
153 (This is now ``+busy``.)
155 #. ``Inconsistent``: This persistent bitmap was not saved to disk
156 correctly, and can no longer be used. It remains in memory to serve as
157 an indicator of failure.
159 (This is now ``+inconsistent``.)
161 These states are directly replaced by the status indicators and should not be
162 used. The difference between ``Frozen`` and ``Locked`` is an implementation
163 detail and should not be relevant to external users.
165 Basic QMP Usage
166 ---------------
168 The primary interface to manipulating bitmap objects is via the QMP
169 interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad
170 overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all
171 QMP commands.
173 Supported Commands
174 ~~~~~~~~~~~~~~~~~~
176 There are six primary bitmap-management API commands:
178 - ``block-dirty-bitmap-add``
179 - ``block-dirty-bitmap-remove``
180 - ``block-dirty-bitmap-clear``
181 - ``block-dirty-bitmap-disable``
182 - ``block-dirty-bitmap-enable``
183 - ``block-dirty-bitmap-merge``
185 And one related query command:
187 - ``query-block``
189 Creation: block-dirty-bitmap-add
190 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
192 `block-dirty-bitmap-add
193 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_:
195 Creates a new bitmap that tracks writes to the specified node. granularity,
196 persistence, and recording state can be adjusted at creation time.
198 .. admonition:: Example
200 to create a new, actively recording persistent bitmap:
202 .. code-block:: QMP
204 -> { "execute": "block-dirty-bitmap-add",
205 "arguments": {
206 "node": "drive0",
207 "name": "bitmap0",
208 "persistent": true,
209 }
210 }
212 <- { "return": {} }
214 - This bitmap will have a default granularity that matches the cluster size of
215 its associated drive, if available, clamped to between [4KiB, 64KiB]. The
216 current default for qcow2 is 64KiB.
218 .. admonition:: Example
220 To create a new, disabled (``-recording``), transient bitmap that tracks
221 changes in 32KiB segments:
223 .. code-block:: QMP
225 -> { "execute": "block-dirty-bitmap-add",
226 "arguments": {
227 "node": "drive0",
228 "name": "bitmap1",
229 "granularity": 32768,
230 "disabled": true
231 }
232 }
234 <- { "return": {} }
236 Deletion: block-dirty-bitmap-remove
237 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
239 `block-dirty-bitmap-remove
240 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_:
242 Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed.
244 - Deleting a bitmap does not impact any other bitmaps attached to the same
245 node, nor does it affect any backups already created from this bitmap or
246 node.
248 - Because bitmaps are only unique to the node to which they are attached, you
249 must specify the node/drive name here, too.
251 - Deleting a persistent bitmap will remove it from the qcow2 file.
253 .. admonition:: Example
255 Remove a bitmap named ``bitmap0`` from node ``drive0``:
257 .. code-block:: QMP
259 -> { "execute": "block-dirty-bitmap-remove",
260 "arguments": {
261 "node": "drive0",
262 "name": "bitmap0"
263 }
264 }
266 <- { "return": {} }
268 Resetting: block-dirty-bitmap-clear
269 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
271 `block-dirty-bitmap-clear
272 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_:
274 Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared.
276 - An incremental backup created from an empty bitmap will copy no data, as if
277 nothing has changed.
279 .. admonition:: Example
281 Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``:
283 .. code-block:: QMP
285 -> { "execute": "block-dirty-bitmap-clear",
286 "arguments": {
287 "node": "drive0",
288 "name": "bitmap0"
289 }
290 }
292 <- { "return": {} }
294 Enabling: block-dirty-bitmap-enable
295 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
297 `block-dirty-bitmap-enable
298 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_:
300 "Enables" a bitmap, setting the ``recording`` bit to true, causing writes to
301 begin being recorded. ``+busy`` bitmaps cannot be enabled.
303 - Bitmaps default to being enabled when created, unless configured otherwise.
305 - Persistent enabled bitmaps will remember their ``+recording`` status on
306 load.
308 .. admonition:: Example
310 To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``:
312 .. code-block:: QMP
314 -> { "execute": "block-dirty-bitmap-enable",
315 "arguments": {
316 "node": "drive0",
317 "name": "bitmap0"
318 }
319 }
321 <- { "return": {} }
323 Enabling: block-dirty-bitmap-disable
324 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
326 `block-dirty-bitmap-disable
327 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_:
329 "Disables" a bitmap, setting the ``recording`` bit to false, causing further
330 writes to begin being ignored. ``+busy`` bitmaps cannot be disabled.
332 .. warning::
334 This is potentially dangerous: QEMU makes no effort to stop any writes if
335 there are disabled bitmaps on a node, and will not mark any disabled bitmaps
336 as ``+inconsistent`` if any such writes do happen. Backups made from such
337 bitmaps will not be able to be used to reconstruct a coherent image.
339 - Disabling a bitmap may be useful for examining which sectors of a disk
340 changed during a specific time period, or for explicit management of
341 differential backup windows.
343 - Persistent disabled bitmaps will remember their ``-recording`` status on
344 load.
346 .. admonition:: Example
348 To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``:
350 .. code-block:: QMP
352 -> { "execute": "block-dirty-bitmap-disable",
353 "arguments": {
354 "node": "drive0",
355 "name": "bitmap0"
356 }
357 }
359 <- { "return": {} }
361 Merging, Copying: block-dirty-bitmap-merge
362 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
364 `block-dirty-bitmap-merge
365 <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_:
367 Merges one or more bitmaps into a target bitmap. For any segment that is dirty
368 in any one source bitmap, the target bitmap will mark that segment dirty.
370 - Merge takes one or more bitmaps as a source and merges them together into a
371 single destination, such that any segment marked as dirty in any source
372 bitmap(s) will be marked dirty in the destination bitmap.
374 - Merge does not create the destination bitmap if it does not exist. A blank
375 bitmap can be created beforehand to achieve the same effect.
377 - The destination is not cleared prior to merge, so subsequent merge
378 operations will continue to cumulatively mark more segments as dirty.
380 - If the merge operation should fail, the destination bitmap is guaranteed to
381 be unmodified. The operation may fail if the source or destination bitmaps
382 are busy, or have different granularities.
384 - Bitmaps can only be merged on the same node. There is only one "node"
385 argument, so all bitmaps must be attached to that same node.
387 - Copy can be achieved by merging from a single source to an empty
388 destination.
390 .. admonition:: Example
392 Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node
393 ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves
394 a copy.
396 .. code-block:: QMP
398 -> { "execute": "block-dirty-bitmap-merge",
399 "arguments": {
400 "node": "drive0",
401 "target": "new_bitmap",
402 "bitmaps": [ "bitmap0" ]
403 }
404 }
406 <- { "return": {} }
408 Querying: query-block
409 ~~~~~~~~~~~~~~~~~~~~~
411 `query-block
412 <qemu-qmp-ref.html#index-query_002dblock>`_:
414 Not strictly a bitmaps command, but will return information about any bitmaps
415 attached to nodes serving as the root for guest devices.
417 - The "inconsistent" bit will not appear when it is false, appearing only when
418 the value is true to indicate there is a problem.
420 .. admonition:: Example
422 Query the block sub-system of QEMU. The following json has trimmed irrelevant
423 keys from the response to highlight only the bitmap-relevant portions of the
424 API. This result highlights a bitmap ``bitmap0`` attached to the root node of
425 device ``drive0``.
427 .. code-block:: QMP
429 -> {
430 "execute": "query-block",
431 "arguments": {}
432 }
434 <- {
435 "return": [ {
436 "dirty-bitmaps": [ {
437 "status": "active",
438 "count": 0,
439 "busy": false,
440 "name": "bitmap0",
441 "persistent": false,
442 "recording": true,
443 "granularity": 65536
444 } ],
445 "device": "drive0",
446 } ]
447 }
449 Bitmap Persistence
450 ------------------
452 As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2
453 files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing
454 ``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to
455 disk.
457 Persistent bitmaps will be automatically loaded into memory upon load, and
458 will be written back to disk upon close. Their usage should be mostly
459 transparent.
461 However, if QEMU does not get a chance to close the file cleanly, the bitmap
462 will be marked as ``+inconsistent`` at next load and considered unsafe to use
463 for any operation. At this point, the only valid operation on such bitmaps is
464 ``block-dirty-bitmap-remove``.
466 Losing a bitmap in this way does not invalidate any existing backups that have
467 been made from this bitmap, but no further backups will be able to be issued
468 for this chain.
470 Transactions
471 ------------
473 Transactions are a QMP feature that allows you to submit multiple QMP commands
474 at once, being guaranteed that they will all succeed or fail atomically,
475 together. The interaction of bitmaps and transactions are demonstrated below.
477 See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference
478 for more details.
480 Justification
481 ~~~~~~~~~~~~~
483 Bitmaps can generally be modified at any time, but certain operations often
484 only make sense when paired directly with other commands. When a VM is paused,
485 it's easy to ensure that no guest writes occur between individual QMP
486 commands. When a VM is running, this is difficult to accomplish with
487 individual QMP commands that may allow guest writes to occur inbetween each
488 command.
490 For example, using only individual QMP commands, we could:
492 #. Boot the VM in a paused state.
493 #. Create a full drive backup of drive0.
494 #. Create a new bitmap attached to drive0, confident that nothing has been
495 written to drive0 in the meantime.
496 #. Resume execution of the VM.
497 #. At a later point, issue incremental backups from ``bitmap0``.
499 At this point, the bitmap and drive backup would be correctly in sync, and
500 incremental backups made from this point forward would be correctly aligned to
501 the full drive backup.
503 This is not particularly useful if we decide we want to start incremental
504 backups after the VM has been running for a while, for which we would want to
505 perform actions such as the following:
507 #. Boot the VM and begin execution.
508 #. Using a single transaction, perform the following operations:
510 - Create ``bitmap0``.
511 - Create a full drive backup of ``drive0``.
513 #. At a later point, issue incremental backups from ``bitmap0``.
515 .. note:: As a consideration, if ``bitmap0`` is created prior to the full
516 drive backup, incremental backups can still be authored from this
517 bitmap, but they will copy extra segments reflecting writes that
518 occurred prior to the backup operation. Transactions allow us to
519 narrow critical points in time to reduce waste, or, in the other
520 direction, to ensure that no segments are omitted.
522 Supported Bitmap Transactions
523 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
525 - ``block-dirty-bitmap-add``
526 - ``block-dirty-bitmap-clear``
527 - ``block-dirty-bitmap-enable``
528 - ``block-dirty-bitmap-disable``
529 - ``block-dirty-bitmap-merge``
531 The usages for these commands are identical to their respective QMP commands,
532 but see the sections below for concrete examples.
534 Incremental Backups - Push Model
535 --------------------------------
537 Incremental backups are simply partial disk images that can be combined with
538 other partial disk images on top of a base image to reconstruct a full backup
539 from the point in time at which the incremental backup was issued.
541 The "Push Model" here references the fact that QEMU is "pushing" the modified
542 blocks out to a destination. We will be using the `drive-backup
543 <qemu-qmp-ref.html#index-drive_002dbackup>`_ and `blockdev-backup
544 <qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP commands to create both
545 full and incremental backups.
547 Both of these commands are jobs, which have their own QMP API for querying and
548 management documented in `Background jobs
549 <qemu-qmp-ref.html#Background-jobs>`_.
551 Example: New Incremental Backup Anchor Point
552 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
554 As outlined in the Transactions - `Justification`_ section, perhaps we want to
555 create a new incremental backup chain attached to a drive.
557 This example creates a new, full backup of "drive0" and accompanies it with a
558 new, empty bitmap that records writes from this point in time forward.
560 .. note:: Any new writes that happen after this command is issued, even while
561 the backup job runs, will be written locally and not to the backup
562 destination. These writes will be recorded in the bitmap
563 accordingly.
565 .. code-block:: QMP
567 -> {
568 "execute": "transaction",
569 "arguments": {
570 "actions": [
571 {
572 "type": "block-dirty-bitmap-add",
573 "data": {
574 "node": "drive0",
575 "name": "bitmap0"
576 }
577 },
578 {
579 "type": "drive-backup",
580 "data": {
581 "device": "drive0",
582 "target": "/path/to/drive0.full.qcow2",
583 "sync": "full",
584 "format": "qcow2"
585 }
586 }
587 ]
588 }
589 }
591 <- { "return": {} }
593 <- {
594 "timestamp": {
595 "seconds": 1555436945,
596 "microseconds": 179620
597 },
598 "data": {
599 "status": "created",
600 "id": "drive0"
601 },
602 "event": "JOB_STATUS_CHANGE"
603 }
605 ...
607 <- {
608 "timestamp": {...},
609 "data": {
610 "device": "drive0",
611 "type": "backup",
612 "speed": 0,
613 "len": 68719476736,
614 "offset": 68719476736
615 },
616 "event": "BLOCK_JOB_COMPLETED"
617 }
619 <- {
620 "timestamp": {...},
621 "data": {
622 "status": "concluded",
623 "id": "drive0"
624 },
625 "event": "JOB_STATUS_CHANGE"
626 }
628 <- {
629 "timestamp": {...},
630 "data": {
631 "status": "null",
632 "id": "drive0"
633 },
634 "event": "JOB_STATUS_CHANGE"
635 }
637 A full explanation of the job transition semantics and the JOB_STATUS_CHANGE
638 event are beyond the scope of this document and will be omitted in all
639 subsequent examples; above, several more events have been omitted for brevity.
641 .. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED
642 except where necessary to illustrate workflow differences.
644 Omitted events and json objects will be represented by ellipses:
645 ``...``
647 Example: Resetting an Incremental Backup Anchor Point
648 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
650 If we want to start a new backup chain with an existing bitmap, we can also
651 use a transaction to reset the bitmap while making a new full backup:
653 .. code-block:: QMP
655 -> {
656 "execute": "transaction",
657 "arguments": {
658 "actions": [
659 {
660 "type": "block-dirty-bitmap-clear",
661 "data": {
662 "node": "drive0",
663 "name": "bitmap0"
664 }
665 },
666 {
667 "type": "drive-backup",
668 "data": {
669 "device": "drive0",
670 "target": "/path/to/drive0.new_full.qcow2",
671 "sync": "full",
672 "format": "qcow2"
673 }
674 }
675 ]
676 }
677 }
679 <- { "return": {} }
681 ...
683 <- {
684 "timestamp": {...},
685 "data": {
686 "device": "drive0",
687 "type": "backup",
688 "speed": 0,
689 "len": 68719476736,
690 "offset": 68719476736
691 },
692 "event": "BLOCK_JOB_COMPLETED"
693 }
695 ...
697 The result of this example is identical to the first, but we clear an existing
698 bitmap instead of adding a new one.
700 .. tip:: In both of these examples, "bitmap0" is tied conceptually to the
701 creation of new, full backups. This relationship is not saved or
702 remembered by QEMU; it is up to the operator or management layer to
703 remember which bitmaps are associated with which backups.
705 Example: First Incremental Backup
706 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
708 #. Create a full backup and sync it to a dirty bitmap using any method:
710 - Either of the two live backup method demonstrated above,
711 - Using QMP commands with the VM paused as in the `Justification`_ section,
712 or
713 - With the VM offline, manually copy the image and start the VM in a paused
714 state, careful to add a new bitmap before the VM begins execution.
716 Whichever method is chosen, let's assume that at the end of this step:
718 - The full backup is named ``drive0.full.qcow2``.
719 - The bitmap we created is named ``bitmap0``, attached to ``drive0``.
721 #. Create a destination image for the incremental backup that utilizes the
722 full backup as a backing image.
724 - Let's assume the new incremental image is named ``drive0.inc0.qcow2``:
726 .. code:: bash
728 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
729 -b drive0.full.qcow2 -F qcow2
731 #. Issue an incremental backup command:
733 .. code-block:: QMP
735 -> {
736 "execute": "drive-backup",
737 "arguments": {
738 "device": "drive0",
739 "bitmap": "bitmap0",
740 "target": "drive0.inc0.qcow2",
741 "format": "qcow2",
742 "sync": "incremental",
743 "mode": "existing"
744 }
745 }
747 <- { "return": {} }
749 ...
751 <- {
752 "timestamp": {...},
753 "data": {
754 "device": "drive0",
755 "type": "backup",
756 "speed": 0,
757 "len": 68719476736,
758 "offset": 68719476736
759 },
760 "event": "BLOCK_JOB_COMPLETED"
761 }
763 ...
765 This copies any blocks modified since the full backup was created into the
766 ``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked
767 ``+busy``. If the operation is successful, ``bitmap0`` will be cleared to
768 reflect the "incremental" backup regimen, which only copies out new changes
769 from each incremental backup.
771 .. note:: Any new writes that occur after the backup operation starts do not
772 get copied to the destination. The backup's "point in time" is when
773 the backup starts, not when it ends. These writes are recorded in a
774 special bitmap that gets re-added to bitmap0 when the backup ends so
775 that the next incremental backup can copy them out.
777 Example: Second Incremental Backup
778 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
780 #. Create a new destination image for the incremental backup that points to
781 the previous one, e.g.: ``drive0.inc1.qcow2``
783 .. code:: bash
785 $ qemu-img create -f qcow2 drive0.inc1.qcow2 \
786 -b drive0.inc0.qcow2 -F qcow2
788 #. Issue a new incremental backup command. The only difference here is that we
789 have changed the target image below.
791 .. code-block:: QMP
793 -> {
794 "execute": "drive-backup",
795 "arguments": {
796 "device": "drive0",
797 "bitmap": "bitmap0",
798 "target": "drive0.inc1.qcow2",
799 "format": "qcow2",
800 "sync": "incremental",
801 "mode": "existing"
802 }
803 }
805 <- { "return": {} }
807 ...
809 <- {
810 "timestamp": {...},
811 "data": {
812 "device": "drive0",
813 "type": "backup",
814 "speed": 0,
815 "len": 68719476736,
816 "offset": 68719476736
817 },
818 "event": "BLOCK_JOB_COMPLETED"
819 }
821 ...
823 Because the first incremental backup from the previous example completed
824 successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here,
825 we use ``bitmap0`` again to create a new incremental backup that targets the
826 previous one, creating a chain of three images:
828 .. admonition:: Diagram
830 .. code:: text
832 +-------------------+ +-------------------+ +-------------------+
833 | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 |
834 +-------------------+ +-------------------+ +-------------------+
836 Each new incremental backup re-synchronizes the bitmap to the latest backup
837 authored, allowing a user to continue to "consume" it to create new backups on
838 top of an existing chain.
840 In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are
841 complete images by themselves, but rely on their backing chain to reconstruct
842 a full image. The dependency terminates with each full backup.
844 Each backup in this chain remains independent, and is unchanged by new entries
845 made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly
846 valid backup of the disk as it was when that backup was issued.
848 Example: Incremental Push Backups without Backing Files
849 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
851 Backup images are best kept off-site, so we often will not have the preceding
852 backups in a chain available to link against. This is not a problem at backup
853 time; we simply do not set the backing image when creating the destination
854 image:
856 #. Create a new destination image with no backing file set. We will need to
857 specify the size of the base image, because the backing file isn't
858 available for QEMU to use to determine it.
860 .. code:: bash
862 $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G
864 .. note:: Alternatively, you can omit ``mode: "existing"`` from the push
865 backup commands to have QEMU create an image without a backing
866 file for you, but you lose control over format options like
867 compatibility and preallocation presets.
869 #. Issue a new incremental backup command. Apart from the new destination
870 image, there is no difference from the last two examples.
872 .. code-block:: QMP
874 -> {
875 "execute": "drive-backup",
876 "arguments": {
877 "device": "drive0",
878 "bitmap": "bitmap0",
879 "target": "drive0.inc2.qcow2",
880 "format": "qcow2",
881 "sync": "incremental",
882 "mode": "existing"
883 }
884 }
886 <- { "return": {} }
888 ...
890 <- {
891 "timestamp": {...},
892 "data": {
893 "device": "drive0",
894 "type": "backup",
895 "speed": 0,
896 "len": 68719476736,
897 "offset": 68719476736
898 },
899 "event": "BLOCK_JOB_COMPLETED"
900 }
902 ...
904 The only difference from the perspective of the user is that you will need to
905 set the backing image when attempting to restore the backup:
907 .. code:: bash
909 $ qemu-img rebase drive0.inc2.qcow2 \
910 -u -b drive0.inc1.qcow2
912 This uses the "unsafe" rebase mode to simply set the backing file to a file
913 that isn't present.
915 It is also possible to use ``--image-opts`` to specify the entire backing
916 chain by hand as an ephemeral property at runtime, but that is beyond the
917 scope of this document.
919 Example: Multi-drive Incremental Backup
920 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
922 Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back
923 both of them up such that the two backups represent the same crash-consistent
924 point in time.
926 #. For each drive, create an empty image:
928 .. code:: bash
930 $ qemu-img create -f qcow2 drive0.full.qcow2 64G
931 $ qemu-img create -f qcow2 drive1.full.qcow2 64G
933 #. Create a full (anchor) backup for each drive, with accompanying bitmaps:
935 .. code-block:: QMP
937 -> {
938 "execute": "transaction",
939 "arguments": {
940 "actions": [
941 {
942 "type": "block-dirty-bitmap-add",
943 "data": {
944 "node": "drive0",
945 "name": "bitmap0"
946 }
947 },
948 {
949 "type": "block-dirty-bitmap-add",
950 "data": {
951 "node": "drive1",
952 "name": "bitmap0"
953 }
954 },
955 {
956 "type": "drive-backup",
957 "data": {
958 "device": "drive0",
959 "target": "/path/to/drive0.full.qcow2",
960 "sync": "full",
961 "format": "qcow2"
962 }
963 },
964 {
965 "type": "drive-backup",
966 "data": {
967 "device": "drive1",
968 "target": "/path/to/drive1.full.qcow2",
969 "sync": "full",
970 "format": "qcow2"
971 }
972 }
973 ]
974 }
975 }
977 <- { "return": {} }
979 ...
981 <- {
982 "timestamp": {...},
983 "data": {
984 "device": "drive0",
985 "type": "backup",
986 "speed": 0,
987 "len": 68719476736,
988 "offset": 68719476736
989 },
990 "event": "BLOCK_JOB_COMPLETED"
991 }
993 ...
995 <- {
996 "timestamp": {...},
997 "data": {
998 "device": "drive1",
999 "type": "backup",
1000 "speed": 0,
1001 "len": 68719476736,
1002 "offset": 68719476736
1003 },
1004 "event": "BLOCK_JOB_COMPLETED"
1005 }
1007 ...
1009 #. Later, create new destination images for each of the incremental backups
1010 that point to their respective full backups:
1012 .. code:: bash
1014 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1015 -b drive0.full.qcow2 -F qcow2
1016 $ qemu-img create -f qcow2 drive1.inc0.qcow2 \
1017 -b drive1.full.qcow2 -F qcow2
1019 #. Issue a multi-drive incremental push backup transaction:
1021 .. code-block:: QMP
1023 -> {
1024 "execute": "transaction",
1025 "arguments": {
1026 "actions": [
1027 {
1028 "type": "drive-backup",
1029 "data": {
1030 "device": "drive0",
1031 "bitmap": "bitmap0",
1032 "format": "qcow2",
1033 "mode": "existing",
1034 "sync": "incremental",
1035 "target": "drive0.inc0.qcow2"
1036 }
1037 },
1038 {
1039 "type": "drive-backup",
1040 "data": {
1041 "device": "drive1",
1042 "bitmap": "bitmap0",
1043 "format": "qcow2",
1044 "mode": "existing",
1045 "sync": "incremental",
1046 "target": "drive1.inc0.qcow2"
1047 }
1048 },
1049 ]
1050 }
1051 }
1053 <- { "return": {} }
1055 ...
1057 <- {
1058 "timestamp": {...},
1059 "data": {
1060 "device": "drive0",
1061 "type": "backup",
1062 "speed": 0,
1063 "len": 68719476736,
1064 "offset": 68719476736
1065 },
1066 "event": "BLOCK_JOB_COMPLETED"
1067 }
1069 ...
1071 <- {
1072 "timestamp": {...},
1073 "data": {
1074 "device": "drive1",
1075 "type": "backup",
1076 "speed": 0,
1077 "len": 68719476736,
1078 "offset": 68719476736
1079 },
1080 "event": "BLOCK_JOB_COMPLETED"
1081 }
1083 ...
1085 Push Backup Errors & Recovery
1086 -----------------------------
1088 In the event of an error that occurs after a push backup job is successfully
1089 launched, either by an individual QMP command or a QMP transaction, the user
1090 will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message,
1091 accompanied by a ``BLOCK_JOB_ERROR`` event.
1093 In the case of a job being cancelled, the user will receive a
1094 ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR
1095 events.
1097 In either failure case, the bitmap used for the failed operation is not
1098 cleared. It will contain all of the dirty bits it did at the start of the
1099 operation, plus any new bits that got marked during the operation.
1101 Effectively, the "point in time" that a bitmap is recording differences
1102 against is kept at the issuance of the last successful incremental backup,
1103 instead of being moved forward to the start of this now-failed backup.
1105 Once the underlying problem is addressed (e.g. more storage space is allocated
1106 on the destination), the incremental backup command can be retried with the
1107 same bitmap.
1109 Example: Individual Failures
1110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112 Incremental Push Backup jobs that fail individually behave simply as
1113 described above. This example demonstrates the single-job failure case:
1115 #. Create a target image:
1117 .. code:: bash
1119 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1120 -b drive0.full.qcow2 -F qcow2
1122 #. Attempt to create an incremental backup via QMP:
1124 .. code-block:: QMP
1126 -> {
1127 "execute": "drive-backup",
1128 "arguments": {
1129 "device": "drive0",
1130 "bitmap": "bitmap0",
1131 "target": "drive0.inc0.qcow2",
1132 "format": "qcow2",
1133 "sync": "incremental",
1134 "mode": "existing"
1135 }
1136 }
1138 <- { "return": {} }
1140 #. Receive a pair of events indicating failure:
1142 .. code-block:: QMP
1144 <- {
1145 "timestamp": {...},
1146 "data": {
1147 "device": "drive0",
1148 "action": "report",
1149 "operation": "write"
1150 },
1151 "event": "BLOCK_JOB_ERROR"
1152 }
1154 <- {
1155 "timestamp": {...},
1156 "data": {
1157 "speed": 0,
1158 "offset": 0,
1159 "len": 67108864,
1160 "error": "No space left on device",
1161 "device": "drive0",
1162 "type": "backup"
1163 },
1164 "event": "BLOCK_JOB_COMPLETED"
1165 }
1167 #. Delete the failed image, and re-create it.
1169 .. code:: bash
1171 $ rm drive0.inc0.qcow2
1172 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \
1173 -b drive0.full.qcow2 -F qcow2
1175 #. Retry the command after fixing the underlying problem, such as
1176 freeing up space on the backup volume:
1178 .. code-block:: QMP
1180 -> {
1181 "execute": "drive-backup",
1182 "arguments": {
1183 "device": "drive0",
1184 "bitmap": "bitmap0",
1185 "target": "drive0.inc0.qcow2",
1186 "format": "qcow2",
1187 "sync": "incremental",
1188 "mode": "existing"
1189 }
1190 }
1192 <- { "return": {} }
1194 #. Receive confirmation that the job completed successfully:
1196 .. code-block:: QMP
1198 <- {
1199 "timestamp": {...},
1200 "data": {
1201 "device": "drive0",
1202 "type": "backup",
1203 "speed": 0,
1204 "len": 67108864,
1205 "offset": 67108864
1206 },
1207 "event": "BLOCK_JOB_COMPLETED"
1208 }
1210 Example: Partial Transactional Failures
1211 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1213 QMP commands like `drive-backup <qemu-qmp-ref.html#index-drive_002dbackup>`_
1214 conceptually only start a job, and so transactions containing these commands
1215 may succeed even if the job it created later fails. This might have surprising
1216 interactions with notions of how a "transaction" ought to behave.
1218 This distinction means that on occasion, a transaction containing such job
1219 launching commands may appear to succeed and return success, but later
1220 individual jobs associated with the transaction may fail. It is possible that
1221 a management application may have to deal with a partial backup failure after
1222 a "successful" transaction.
1224 If multiple backup jobs are specified in a single transaction, if one of those
1225 jobs fails, it will not interact with the other backup jobs in any way by
1226 default. The job(s) that succeeded will clear the dirty bitmap associated with
1227 the operation, but the job(s) that failed will not. It is therefore not safe
1228 to delete any incremental backups that were created successfully in this
1229 scenario, even though others failed.
1231 This example illustrates a transaction with two backup jobs, where one fails
1232 and one succeeds:
1234 #. Issue the transaction to start a backup of both drives.
1236 .. code-block:: QMP
1238 -> {
1239 "execute": "transaction",
1240 "arguments": {
1241 "actions": [
1242 {
1243 "type": "drive-backup",
1244 "data": {
1245 "device": "drive0",
1246 "bitmap": "bitmap0",
1247 "format": "qcow2",
1248 "mode": "existing",
1249 "sync": "incremental",
1250 "target": "drive0.inc0.qcow2"
1251 }
1252 },
1253 {
1254 "type": "drive-backup",
1255 "data": {
1256 "device": "drive1",
1257 "bitmap": "bitmap0",
1258 "format": "qcow2",
1259 "mode": "existing",
1260 "sync": "incremental",
1261 "target": "drive1.inc0.qcow2"
1262 }
1263 }]
1264 }
1265 }
1267 #. Receive notice that the Transaction was accepted, and jobs were
1268 launched:
1270 .. code-block:: QMP
1272 <- { "return": {} }
1274 #. Receive notice that the first job has completed:
1276 .. code-block:: QMP
1278 <- {
1279 "timestamp": {...},
1280 "data": {
1281 "device": "drive0",
1282 "type": "backup",
1283 "speed": 0,
1284 "len": 67108864,
1285 "offset": 67108864
1286 },
1287 "event": "BLOCK_JOB_COMPLETED"
1288 }
1290 #. Receive notice that the second job has failed:
1292 .. code-block:: QMP
1294 <- {
1295 "timestamp": {...},
1296 "data": {
1297 "device": "drive1",
1298 "action": "report",
1299 "operation": "read"
1300 },
1301 "event": "BLOCK_JOB_ERROR"
1302 }
1304 ...
1306 <- {
1307 "timestamp": {...},
1308 "data": {
1309 "speed": 0,
1310 "offset": 0,
1311 "len": 67108864,
1312 "error": "Input/output error",
1313 "device": "drive1",
1314 "type": "backup"
1315 },
1316 "event": "BLOCK_JOB_COMPLETED"
1317 }
1319 At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and
1320 must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be
1321 deleted. If a VM-wide incremental backup of all drives at a point-in-time is
1322 to be made, new backups for both drives will need to be made, taking into
1323 account that a new incremental backup for drive0 needs to be based on top of
1324 ``drive0.inc0.qcow2``.
1326 For this example, an incremental backup for ``drive0`` was created, but not
1327 for ``drive1``. The last VM-wide crash-consistent backup that is available in
1328 this case is the full backup:
1330 .. code:: text
1332 [drive0.full.qcow2] <-- [drive0.inc0.qcow2]
1333 [drive1.full.qcow2]
1335 To repair this, issue a new incremental backup across both drives. The result
1336 will be backup chains that resemble the following:
1338 .. code:: text
1340 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2]
1341 [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2]
1343 Example: Grouped Completion Mode
1344 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1346 While jobs launched by transactions normally complete or fail individually,
1347 it's possible to instruct them to complete or fail together as a group. QMP
1348 transactions take an optional properties structure that can affect the
1349 behavior of the transaction.
1351 The ``completion-mode`` transaction property can be either ``individual``
1352 which is the default legacy behavior described above, or ``grouped``, detailed
1353 below.
1355 In ``grouped`` completion mode, no jobs will report success until all jobs are
1356 ready to report success. If any job fails, all other jobs will be cancelled.
1358 Regardless of if a participating incremental backup job failed or was
1359 cancelled, their associated bitmaps will all be held at their existing
1360 points-in-time, as in individual failure cases.
1362 Here's the same multi-drive backup scenario from `Example: Partial
1363 Transactional Failures`_, but with the ``grouped`` completion-mode property
1364 applied:
1366 #. Issue the multi-drive incremental backup transaction:
1368 .. code-block:: QMP
1370 -> {
1371 "execute": "transaction",
1372 "arguments": {
1373 "properties": {
1374 "completion-mode": "grouped"
1375 },
1376 "actions": [
1377 {
1378 "type": "drive-backup",
1379 "data": {
1380 "device": "drive0",
1381 "bitmap": "bitmap0",
1382 "format": "qcow2",
1383 "mode": "existing",
1384 "sync": "incremental",
1385 "target": "drive0.inc0.qcow2"
1386 }
1387 },
1388 {
1389 "type": "drive-backup",
1390 "data": {
1391 "device": "drive1",
1392 "bitmap": "bitmap0",
1393 "format": "qcow2",
1394 "mode": "existing",
1395 "sync": "incremental",
1396 "target": "drive1.inc0.qcow2"
1397 }
1398 }]
1399 }
1400 }
1402 #. Receive notice that the Transaction was accepted, and jobs were launched:
1404 .. code-block:: QMP
1406 <- { "return": {} }
1408 #. Receive notification that the backup job for ``drive1`` has failed:
1410 .. code-block:: QMP
1412 <- {
1413 "timestamp": {...},
1414 "data": {
1415 "device": "drive1",
1416 "action": "report",
1417 "operation": "read"
1418 },
1419 "event": "BLOCK_JOB_ERROR"
1420 }
1422 <- {
1423 "timestamp": {...},
1424 "data": {
1425 "speed": 0,
1426 "offset": 0,
1427 "len": 67108864,
1428 "error": "Input/output error",
1429 "device": "drive1",
1430 "type": "backup"
1431 },
1432 "event": "BLOCK_JOB_COMPLETED"
1433 }
1435 #. Receive notification that the job for ``drive0`` has been cancelled:
1437 .. code-block:: QMP
1439 <- {
1440 "timestamp": {...},
1441 "data": {
1442 "device": "drive0",
1443 "type": "backup",
1444 "speed": 0,
1445 "len": 67108864,
1446 "offset": 16777216
1447 },
1448 "event": "BLOCK_JOB_CANCELLED"
1449 }
1451 At the conclusion of *this* example, both jobs have been aborted due to a
1452 failure. Both destination images should be deleted and are no longer of use.
1454 The transaction as a whole can simply be re-issued at a later time.
1456 .. raw:: html
1458 <!--
1459 The FreeBSD Documentation License
1461 Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML,
1462 PDF, PostScript, RTF and so forth) with or without modification, are
1463 permitted provided that the following conditions are met:
1465 Redistributions of source code (ReST) must retain the above copyright notice,
1466 this list of conditions and the following disclaimer of this file unmodified.
1468 Redistributions in compiled form (transformed to other DTDs, converted to
1469 PDF, PostScript, RTF and other formats) must reproduce the above copyright
1470 notice, this list of conditions and the following disclaimer in the
1471 documentation and/or other materials provided with the distribution.
1473 THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
1474 IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
1475 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
1476 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
1477 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
1478 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
1479 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
1480 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
1481 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
1482 ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF
1483 THE POSSIBILITY OF SUCH DAMAGE.
1484 -->