docs/COLO-FT.txt

   1 COarse-grained LOck-stepping Virtual Machines for Non-stop Service
   2 ----------------------------------------
   3 Copyright (c) 2016 Intel Corporation
   4 Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD.
   5 Copyright (c) 2016 Fujitsu, Corp.
   6
   7 This work is licensed under the terms of the GNU GPL, version 2 or later.
   8 See the COPYING file in the top-level directory.
   9
  10 This document gives an overview of COLO's design and how to use it.
  11
  12 == Background ==
  13 Virtual machine (VM) replication is a well known technique for providing
  14 application-agnostic software-implemented hardware fault tolerance,
  15 also known as "non-stop service".
  16
  17 COLO (COarse-grained LOck-stepping) is a high availability solution.
  18 Both primary VM (PVM) and secondary VM (SVM) run in parallel. They receive the
  19 same request from client, and generate response in parallel too.
  20 If the response packets from PVM and SVM are identical, they are released
  21 immediately. Otherwise, a VM checkpoint (on demand) is conducted.
  22
  23 == Architecture ==
  24
  25 The architecture of COLO is shown in the diagram below.
  26 It consists of a pair of networked physical nodes:
  27 The primary node running the PVM, and the secondary node running the SVM
  28 to maintain a valid replica of the PVM.
  29 PVM and SVM execute in parallel and generate output of response packets for
  30 client requests according to the application semantics.
  31
  32 The incoming packets from the client or external network are received by the
  33 primary node, and then forwarded to the secondary node, so that both the PVM
  34 and the SVM are stimulated with the same requests.
  35
  36 COLO receives the outbound packets from both the PVM and SVM and compares them
  37 before allowing the output to be sent to clients.
  38
  39 The SVM is qualified as a valid replica of the PVM, as long as it generates
  40 identical responses to all client requests. Once the differences in the outputs
  41 are detected between the PVM and SVM, COLO withholds transmission of the
  42 outbound packets until it has successfully synchronized the PVM state to the SVM.
  43
  44   Primary Node                                                            Secondary Node
  45 +------------+  +-----------------------+       +------------------------+  +------------+
  46 |            |  |       HeartBeat       +<----->+       HeartBeat        |  |            |
  47 | Primary VM |  +-----------+-----------+       +-----------+------------+  |Secondary VM|
  48 |            |              |                               |               |            |
  49 |            |  +-----------|-----------+       +-----------|------------+  |            |
  50 |            |  |QEMU   +---v----+      |       |QEMU  +----v---+        |  |            |
  51 |            |  |       |Failover|      |       |      |Failover|        |  |            |
  52 |            |  |       +--------+      |       |      +--------+        |  |            |
  53 |            |  |   +---------------+   |       |   +---------------+    |  |            |
  54 |            |  |   | VM Checkpoint +-------------->+ VM Checkpoint |    |  |            |
  55 |            |  |   +---------------+   |       |   +---------------+    |  |            |
  56 |Requests<--------------------------\ /-----------------\ /--------------------->Requests|
  57 |            |  |                   ^ ^ |       |       | |              |  |            |
  58 |Responses+---------------------\ /-|-|------------\ /-------------------------+Responses|
  59 |            |  |               | | | | |       |  | |  | |              |  |            |
  60 |            |  | +-----------+ | | | | |       |  | |  | | +----------+ |  |            |
  61 |            |  | | COLO disk | | | | | |       |  | |  | | | COLO disk| |  |            |
  62 |            |  | |   Manager +---------------------------->| Manager  | |  |            |
  63 |            |  | ++----------+ v v | | |       |  | v  v | +---------++ |  |            |
  64 |            |  |  |+-----------+-+-+-++|       | ++-+--+-+---------+ |  |  |            |
  65 |            |  |  ||   COLO Proxy     ||       | |   COLO Proxy    | |  |  |            |
  66 |            |  |  || (compare packet  ||       | |(adjust sequence | |  |  |            |
  67 |            |  |  ||and mirror packet)||       | |    and ACK)     | |  |  |            |
  68 |            |  |  |+------------+---+-+|       | +-----------------+ |  |  |            |
  69 +------------+  +-----------------------+       +------------------------+  +------------+
  70 +------------+     |             |   |                                |     +------------+
  71 | VM Monitor |     |             |   |                                |     | VM Monitor |
  72 +------------+     |             |   |                                |     +------------+
  73 +---------------------------------------+       +----------------------------------------+
  74 |   Kernel         |             |   |  |       |   Kernel            |                  |
  75 +---------------------------------------+       +----------------------------------------+
  76                    |             |   |                                |
  77     +--------------v+  +---------v---+--+       +------------------+ +v-------------+
  78     |   Storage     |  |External Network|       | External Network | |   Storage    |
  79     +---------------+  +----------------+       +------------------+ +--------------+
  80
  81
  82 == Components introduction ==
  83
  84 You can see there are several components in COLO's diagram of architecture.
  85 Their functions are described below.
  86
  87 HeartBeat:
  88 Runs on both the primary and secondary nodes, to periodically check platform
  89 availability. When the primary node suffers a hardware fail-stop failure,
  90 the heartbeat stops responding, the secondary node will trigger a failover
  91 as soon as it determines the absence.
  92
  93 COLO disk Manager:
  94 When primary VM writes data into image, the colo disk manger captures this data
  95 and sends it to secondary VM's which makes sure the context of secondary VM's
  96 image is consistent with the context of primary VM 's image.
  97 For more details, please refer to docs/block-replication.txt.
  98
  99 Checkpoint/Failover Controller:
 100 Modifications of save/restore flow to realize continuous migration,
 101 to make sure the state of VM in Secondary side is always consistent with VM in
 102 Primary side.
 103
 104 COLO Proxy:
 105 Delivers packets to Primary and Seconday, and then compare the responses from
 106 both side. Then decide whether to start a checkpoint according to some rules.
 107 Please refer to docs/colo-proxy.txt for more informations.
 108
 109 Note:
 110 HeartBeat has not been implemented yet, so you need to trigger failover process
 111 by using 'x-colo-lost-heartbeat' command.
 112
 113 == Test procedure ==
 114 1. Startup qemu
 115 Primary:
 116 # qemu-kvm -enable-kvm -m 2048 -smp 2 -qmp stdio -vnc :7 -name primary \
 117   -device piix3-usb-uhci \
 118   -device usb-tablet -netdev tap,id=hn0,vhost=off \
 119   -device virtio-net-pci,id=net-pci0,netdev=hn0 \
 120   -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
 121          children.0.file.filename=1.raw,\
 122          children.0.driver=raw -S
 123 Secondary:
 124 # qemu-kvm -enable-kvm -m 2048 -smp 2 -qmp stdio -vnc :7 -name secondary \
 125   -device piix3-usb-uhci \
 126   -device usb-tablet -netdev tap,id=hn0,vhost=off \
 127   -device virtio-net-pci,id=net-pci0,netdev=hn0 \
 128   -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-name=node0 \
 129   -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
 130          file.driver=qcow2,top-id=active-disk0,\
 131          file.file.filename=/mnt/ramfs/active_disk.img,\
 132          file.backing.driver=qcow2,\
 133          file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
 134          file.backing.backing=secondary-disk0 \
 135   -incoming tcp:0:8888
 136
 137 2. On Secondary VM's QEMU monitor, issue command
 138 {'execute':'qmp_capabilities'}
 139 { 'execute': 'nbd-server-start',
 140   'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port': '8889'} } }
 141 }
 142 {'execute': 'nbd-server-add', 'arguments': {'device': 'secondeary-disk0', 'writable': true } }
 143
 144 Note:
 145   a. The qmp command nbd-server-start and nbd-server-add must be run
 146      before running the qmp command migrate on primary QEMU
 147   b. Active disk, hidden disk and nbd target's length should be the
 148      same.
 149   c. It is better to put active disk and hidden disk in ramdisk.
 150
 151 3. On Primary VM's QEMU monitor, issue command:
 152 {'execute':'qmp_capabilities'}
 153 { 'execute': 'human-monitor-command',
 154   'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.port=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
 155 { 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } }
 156 { 'execute': 'migrate-set-capabilities',
 157       'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
 158 { 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
 159
 160   Note:
 161   a. There should be only one NBD Client for each primary disk.
 162   b. xx.xx.xx.xx is the secondary physical machine's hostname or IP
 163   c. The qmp command line must be run after running qmp command line in
 164      secondary qemu.
 165
 166 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
 167 You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }'
 168 to change the checkpoint period time
 169
 170 5. Failover test
 171 You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
 172 monitor at the same time, then SVM will failover and client will not detect this
 173 change.
 174
 175 Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
 176 issue block related command to stop block replication.
 177 Primary:
 178   Remove the nbd child from the quorum:
 179   { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
 180   { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
 181   Note: there is no qmp command to remove the blockdev now
 182
 183 Secondary:
 184   The primary host is down, so we should do the following thing:
 185   { 'execute': 'nbd-server-stop' }
 186
 187 == TODO ==
 188 1. Support continuous VM replication.
 189 2. Support shared storage.
 190 3. Develop the heartbeat part.
 191 4. Reduce checkpoint VM’s downtime while doing checkpoint.