4 :Copyright: 2014 Virtual Open Systems Sarl.
5 :Licence: This work is licensed under the terms of the GNU GPL,
6 version 2 or later. See the COPYING file in the top-level
9 .. contents:: Table of Contents
14 This protocol is aiming to complement the ``ioctl`` interface used to
15 control the vhost implementation in the Linux kernel. It implements
16 the control plane needed to establish virtqueue sharing with a user
17 space process on the same host. It uses communication over a Unix
18 domain socket to share file descriptors in the ancillary data of the
21 The protocol defines 2 sides of the communication, *master* and
22 *slave*. *Master* is the application that shares its virtqueues, in
23 our case QEMU. *Slave* is the consumer of the virtqueues.
25 In the current implementation QEMU is the *master*, and the *slave* is
26 the external process consuming the virtio queues, for example a
27 software Ethernet switch running in user space, such as Snabbswitch,
28 or a block device backend processing read & write to a virtual
29 disk. In order to facilitate interoperability between various backend
30 implementations, it is recommended to follow the :ref:`Backend program
31 conventions <backend_conventions>`.
33 *Master* and *slave* can be either a client (i.e. connecting) or
34 server (listening) in the socket communication.
39 .. Note:: All numbers are in the machine native byte order.
41 A vhost-user message consists of 3 header fields and a payload.
43 +---------+-------+------+---------+
44 | request | flags | size | payload |
45 +---------+-------+------+---------+
50 :request: 32-bit type of the request
52 :flags: 32-bit bit field
54 - Lower 2 bits are the version (currently 0x01)
55 - Bit 2 is the reply flag - needs to be sent on each reply from the slave
56 - Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
59 :size: 32-bit size of the payload
64 Depending on the request type, **payload** can be:
66 A single 64-bit integer
67 ^^^^^^^^^^^^^^^^^^^^^^^
73 :u64: a 64-bit unsigned integer
75 A vring state description
76 ^^^^^^^^^^^^^^^^^^^^^^^^^
82 :index: a 32-bit index
86 A vring address description
87 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
89 +-------+-------+------+------------+------+-----------+-----+
90 | index | flags | size | descriptor | used | available | log |
91 +-------+-------+------+------------+------+-----------+-----+
93 :index: a 32-bit vring index
95 :flags: a 32-bit vring flags
97 :descriptor: a 64-bit ring address of the vring descriptor table
99 :used: a 64-bit ring address of the vring used ring
101 :available: a 64-bit ring address of the vring available ring
103 :log: a 64-bit guest address for logging
105 Note that a ring address is an IOVA if ``VIRTIO_F_IOMMU_PLATFORM`` has
106 been negotiated. Otherwise it is a user address.
108 Memory regions description
109 ^^^^^^^^^^^^^^^^^^^^^^^^^^
111 +-------------+---------+---------+-----+---------+
112 | num regions | padding | region0 | ... | region7 |
113 +-------------+---------+---------+-----+---------+
115 :num regions: a 32-bit number of regions
121 +---------------+------+--------------+-------------+
122 | guest address | size | user address | mmap offset |
123 +---------------+------+--------------+-------------+
125 :guest address: a 64-bit guest address of the region
129 :user address: a 64-bit user address
131 :mmap offset: 64-bit offset where region starts in the mapped memory
136 +----------+------------+
137 | log size | log offset |
138 +----------+------------+
140 :log size: size of area used for logging
142 :log offset: offset from start of supplied file descriptor where
143 logging starts (i.e. where guest address 0 would be
149 +------+------+--------------+-------------------+------+
150 | iova | size | user address | permissions flags | type |
151 +------+------+--------------+-------------------+------+
153 :iova: a 64-bit I/O virtual address programmed by the guest
157 :user address: a 64-bit user address
159 :permissions flags: an 8-bit value:
163 - 3: Read/Write access
165 :type: an 8-bit IOTLB message type:
168 - 3: IOTLB invalidate
169 - 4: IOTLB access fail
171 Virtio device config space
172 ^^^^^^^^^^^^^^^^^^^^^^^^^^
174 +--------+------+-------+---------+
175 | offset | size | flags | payload |
176 +--------+------+-------+---------+
178 :offset: a 32-bit offset of virtio device's configuration space
180 :size: a 32-bit configuration space access size in bytes
182 :flags: a 32-bit value:
183 - 0: Vhost master messages used for writeable fields
184 - 1: Vhost master messages used for live migration
186 :payload: Size bytes array holding the contents of the virtio
187 device's configuration space
189 Vring area description
190 ^^^^^^^^^^^^^^^^^^^^^^
192 +-----+------+--------+
193 | u64 | size | offset |
194 +-----+------+--------+
196 :u64: a 64-bit integer contains vring index and flags
198 :size: a 64-bit size of this area
200 :offset: a 64-bit offset of this area from the start of the
201 supplied file descriptor
206 +-----------+-------------+------------+------------+
207 | mmap size | mmap offset | num queues | queue size |
208 +-----------+-------------+------------+------------+
210 :mmap size: a 64-bit size of area to track inflight I/O
212 :mmap offset: a 64-bit offset of this area from the start
213 of the supplied file descriptor
215 :num queues: a 16-bit number of virtqueues
217 :queue size: a 16-bit size of virtqueues
222 In QEMU the vhost-user message is implemented with the following struct:
226 typedef struct VhostUserMsg {
227 VhostUserRequest request;
232 struct vhost_vring_state state;
233 struct vhost_vring_addr addr;
234 VhostUserMemory memory;
236 struct vhost_iotlb_msg iotlb;
237 VhostUserConfig config;
238 VhostUserVringArea area;
239 VhostUserInflight inflight;
241 } QEMU_PACKED VhostUserMsg;
246 The protocol for vhost-user is based on the existing implementation of
247 vhost for the Linux Kernel. Most messages that can be sent via the
248 Unix domain socket implementing vhost-user have an equivalent ioctl to
249 the kernel implementation.
251 The communication consists of *master* sending message requests and
252 *slave* sending message replies. Most of the requests don't require
253 replies. Here is a list of the ones that do:
255 * ``VHOST_USER_GET_FEATURES``
256 * ``VHOST_USER_GET_PROTOCOL_FEATURES``
257 * ``VHOST_USER_GET_VRING_BASE``
258 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
259 * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
263 :ref:`REPLY_ACK <reply_ack>`
264 The section on ``REPLY_ACK`` protocol extension.
266 There are several messages that the master sends with file descriptors passed
267 in the ancillary data:
269 * ``VHOST_USER_SET_MEM_TABLE``
270 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
271 * ``VHOST_USER_SET_LOG_FD``
272 * ``VHOST_USER_SET_VRING_KICK``
273 * ``VHOST_USER_SET_VRING_CALL``
274 * ``VHOST_USER_SET_VRING_ERR``
275 * ``VHOST_USER_SET_SLAVE_REQ_FD``
276 * ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
278 If *master* is unable to send the full message or receives a wrong
279 reply it will close the connection. An optional reconnection mechanism
282 Any protocol extensions are gated by protocol feature bits, which
283 allows full backwards compatibility on both master and slave. As
284 older slaves don't support negotiating protocol features, a feature
285 bit was dedicated for this purpose::
287 #define VHOST_USER_F_PROTOCOL_FEATURES 30
289 Starting and stopping rings
290 ---------------------------
292 Client must only process each ring when it is started.
294 Client must only pass data between the ring and the backend, when the
297 If ring is started but disabled, client must process the ring without
298 talking to the backend.
300 For example, for a networking device, in the disabled state client
301 must not supply any new RX packets, but must process and discard any
304 If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the
305 ring is initialized in an enabled state.
307 If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is
308 initialized in a disabled state. Client must not pass data to/from the
309 backend until ring is enabled by ``VHOST_USER_SET_VRING_ENABLE`` with
310 parameter 1, or after it has been disabled by
311 ``VHOST_USER_SET_VRING_ENABLE`` with parameter 0.
313 Each ring is initialized in a stopped state, client must not process
314 it until ring is started, or after it has been stopped.
316 Client must start ring upon receiving a kick (that is, detecting that
317 file descriptor is readable) on the descriptor specified by
318 ``VHOST_USER_SET_VRING_KICK``, and stop ring upon receiving
319 ``VHOST_USER_GET_VRING_BASE``.
321 While processing the rings (whether they are enabled or not), client
322 must support changing some configuration aspects on the fly.
324 Multiple queue support
325 ----------------------
327 Multiple queue support allows the slave to advertise the maximum number of
328 queues. This is treated as a protocol extension, hence the slave has to
329 implement protocol features first. The multiple queues feature is supported
330 only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ`` (bit 0) is set.
332 The max number of queues the slave supports can be queried with message
333 ``VHOST_USER_GET_QUEUE_NUM``. Master should stop when the number of requested
334 queues is bigger than that.
336 As all queues share one connection, the master uses a unique index for each
337 queue in the sent message to identify a specified queue.
339 The master enables queues by sending message ``VHOST_USER_SET_VRING_ENABLE``.
340 vhost-user-net has historically automatically enabled the first queue pair.
345 During live migration, the master may need to track the modifications
346 the slave makes to the memory mapped regions. The client should mark
347 the dirty pages in a log. Once it complies to this logging, it may
348 declare the ``VHOST_F_LOG_ALL`` vhost feature.
350 To start/stop logging of data/used ring writes, server may send
351 messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
352 ``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
353 flags set to 1/0, respectively.
355 All the modifications to memory pointed by vring "descriptor" should
356 be marked. Modifications to "used" vring should be marked if
357 ``VHOST_VRING_F_LOG`` is part of ring's flags.
359 Dirty pages are of size::
361 #define VHOST_LOG_PAGE 0x1000
363 The log memory fd is provided in the ancillary data of
364 ``VHOST_USER_SET_LOG_BASE`` message when the slave has
365 ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.
367 The size of the log is supplied as part of ``VhostUserMsg`` which
368 should be large enough to cover all known guest addresses. Log starts
369 at the supplied offset in the supplied file descriptor. The log
370 covers from address 0 to the maximum of guest regions. In pseudo-code,
371 to mark page at ``addr`` as dirty::
373 page = addr / VHOST_LOG_PAGE
374 log[page / 8] |= 1 << page % 8
376 Where ``addr`` is the guest physical address.
378 Use atomic operations, as the log may be concurrently manipulated.
380 Note that when logging modifications to the used ring (when
381 ``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
382 be used to calculate the log offset: the write to first byte of the
383 used ring is logged at this offset from log start. Also note that this
384 value might be outside the legal guest physical address range
385 (i.e. does not have to be covered by the ``VhostUserMemory`` table), but
386 the bit offset of the last byte of the ring must fall within the size
387 supplied by ``VhostUserLog``.
389 ``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
390 ancillary data, it may be used to inform the master that the log has
393 Once the source has finished migration, rings will be stopped by the
394 source. No further update must be done before rings are restarted.
396 In postcopy migration the slave is started before all the memory has
397 been received from the source host, and care must be taken to avoid
398 accessing pages that have yet to be received. The slave opens a
399 'userfault'-fd and registers the memory with it; this fd is then
400 passed back over to the master. The master services requests on the
401 userfaultfd for pages that are accessed and when the page is available
402 it performs WAKE ioctl's on the userfaultfd to wake the stalled
403 slave. The client indicates support for this via the
404 ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.
409 The master sends a list of vhost memory regions to the slave using the
410 ``VHOST_USER_SET_MEM_TABLE`` message. Each region has two base
411 addresses: a guest address and a user address.
413 Messages contain guest addresses and/or user addresses to reference locations
414 within the shared memory. The mapping of these addresses works as follows.
416 User addresses map to the vhost memory region containing that user address.
418 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:
420 * Guest addresses map to the vhost memory region containing that guest
423 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:
425 * Guest addresses are also called I/O virtual addresses (IOVAs). They are
426 translated to user addresses via the IOTLB.
428 * The vhost memory region guest address is not used.
433 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
434 master sends IOTLB entries update & invalidation by sending
435 ``VHOST_USER_IOTLB_MSG`` requests to the slave with a ``struct
436 vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
437 has to be filled with the update message type (2), the I/O virtual
438 address, the size, the user virtual address, and the permissions
439 flags. Addresses and size must be within vhost memory regions set via
440 the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
441 ``iotlb`` payload has to be filled with the invalidation message type
442 (3), the I/O virtual address and the size. On success, the slave is
443 expected to reply with a zero payload, non-zero otherwise.
445 The slave relies on the slave communcation channel (see :ref:`Slave
446 communication <slave_communication>` section below) to send IOTLB miss
447 and access failure events, by sending ``VHOST_USER_SLAVE_IOTLB_MSG``
448 requests to the master with a ``struct vhost_iotlb_msg`` as
449 payload. For miss events, the iotlb payload has to be filled with the
450 miss message type (1), the I/O virtual address and the permissions
451 flags. For access failure event, the iotlb payload has to be filled
452 with the access failure message type (4), the I/O virtual address and
453 the permissions flags. For synchronization purpose, the slave may
454 rely on the reply-ack feature, so the master may send a reply when
455 operation is completed if the reply-ack feature is negotiated and
456 slaves requests a reply. For miss events, completed operation means
457 either master sent an update message containing the IOTLB entry
458 containing requested address and permission, or master sent nothing if
459 the IOTLB miss message is invalid (invalid IOVA or permission).
461 The master isn't expected to take the initiative to send IOTLB update
462 messages, as the slave sends IOTLB miss messages for the guest virtual
463 memory areas it needs to access.
465 .. _slave_communication:
470 An optional communication channel is provided if the slave declares
471 ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` protocol feature, to allow the
472 slave to make requests to the master.
474 The fd is provided via ``VHOST_USER_SET_SLAVE_REQ_FD`` ancillary data.
476 A slave may then send ``VHOST_USER_SLAVE_*`` messages to the master
477 using this fd communication channel.
479 If ``VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD`` protocol feature is
480 negotiated, slave can send file descriptors (at most 8 descriptors in
481 each message) to master via ancillary data using this fd communication
484 Inflight I/O tracking
485 ---------------------
487 To support reconnecting after restart or crash, slave may need to
488 resubmit inflight I/Os. If virtqueue is processed in order, we can
489 easily achieve that by getting the inflight descriptors from
490 descriptor table (split virtqueue) or descriptor ring (packed
491 virtqueue). However, it can't work when we process descriptors
492 out-of-order because some entries which store the information of
493 inflight descriptors in available ring (split virtqueue) or descriptor
494 ring (packed virtqueue) might be overrided by new entries. To solve
495 this problem, slave need to allocate an extra buffer to store this
496 information of inflight descriptors and share it with master for
497 persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
498 ``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
499 between master and slave. And the format of this buffer is described
502 +---------------+---------------+-----+---------------+
503 | queue0 region | queue1 region | ... | queueN region |
504 +---------------+---------------+-----+---------------+
506 N is the number of available virtqueues. Slave could get it from num
507 queues field of ``VhostUserInflight``.
509 For split virtqueue, queue region can be implemented as:
513 typedef struct DescStateSplit {
514 /* Indicate whether this descriptor is inflight or not.
515 * Only available for head-descriptor. */
521 /* Maintain a list for the last batch of used descriptors.
522 * Only available when batching is used for submitting */
525 /* Used to preserve the order of fetching available descriptors.
526 * Only available for head-descriptor. */
530 typedef struct QueueRegionSplit {
531 /* The feature flags of this region. Now it's initialized to 0. */
534 /* The version of this region. It's 1 currently.
535 * Zero value indicates an uninitialized buffer */
538 /* The size of DescStateSplit array. It's equal to the virtqueue
539 * size. Slave could get it from queue size field of VhostUserInflight. */
542 /* The head of list that track the last batch of used descriptors. */
543 uint16_t last_batch_head;
545 /* Store the idx value of used ring */
548 /* Used to track the state of each descriptor in descriptor table */
549 DescStateSplit desc[0];
552 To track inflight I/O, the queue region should be processed as follows:
554 When receiving available buffers from the driver:
556 #. Get the next available head-descriptor index from available ring, ``i``
558 #. Set ``desc[i].counter`` to the value of global counter
560 #. Increase global counter by 1
562 #. Set ``desc[i].inflight`` to 1
564 When supplying used buffers to the driver:
566 1. Get corresponding used head-descriptor index, i
568 2. Set ``desc[i].next`` to ``last_batch_head``
570 3. Set ``last_batch_head`` to ``i``
572 #. Steps 1,2,3 may be performed repeatedly if batching is possible
574 #. Increase the ``idx`` value of used ring by the size of the batch
576 #. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0
578 #. Set ``used_idx`` to the ``idx`` value of used ring
582 #. If the value of ``used_idx`` does not match the ``idx`` value of
583 used ring (means the inflight field of ``DescStateSplit`` entries in
584 last batch may be incorrect),
586 a. Subtract the value of ``used_idx`` from the ``idx`` value of
587 used ring to get last batch size of ``DescStateSplit`` entries
589 #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
590 list which starts from ``last_batch_head``
592 #. Set ``used_idx`` to the ``idx`` value of used ring
594 #. Resubmit inflight ``DescStateSplit`` entries in order of their
597 For packed virtqueue, queue region can be implemented as:
601 typedef struct DescStatePacked {
602 /* Indicate whether this descriptor is inflight or not.
603 * Only available for head-descriptor. */
609 /* Link to the next free entry */
612 /* Link to the last entry of descriptor list.
613 * Only available for head-descriptor. */
616 /* The length of descriptor list.
617 * Only available for head-descriptor. */
620 /* Used to preserve the order of fetching available descriptors.
621 * Only available for head-descriptor. */
627 /* The descriptor flags */
630 /* The buffer length */
633 /* The buffer address */
637 typedef struct QueueRegionPacked {
638 /* The feature flags of this region. Now it's initialized to 0. */
641 /* The version of this region. It's 1 currently.
642 * Zero value indicates an uninitialized buffer */
645 /* The size of DescStatePacked array. It's equal to the virtqueue
646 * size. Slave could get it from queue size field of VhostUserInflight. */
649 /* The head of free DescStatePacked entry list */
652 /* The old head of free DescStatePacked entry list */
653 uint16_t old_free_head;
655 /* The used index of descriptor ring */
658 /* The old used index of descriptor ring */
659 uint16_t old_used_idx;
661 /* Device ring wrap counter */
662 uint8_t used_wrap_counter;
664 /* The old device ring wrap counter */
665 uint8_t old_used_wrap_counter;
670 /* Used to track the state of each descriptor fetched from descriptor ring */
671 DescStatePacked desc[0];
674 To track inflight I/O, the queue region should be processed as follows:
676 When receiving available buffers from the driver:
678 #. Get the next available descriptor entry from descriptor ring, ``d``
680 #. If ``d`` is head descriptor,
682 a. Set ``desc[old_free_head].num`` to 0
684 #. Set ``desc[old_free_head].counter`` to the value of global counter
686 #. Increase global counter by 1
688 #. Set ``desc[old_free_head].inflight`` to 1
690 #. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
693 #. Increase ``desc[old_free_head].num`` by 1
695 #. Set ``desc[free_head].addr``, ``desc[free_head].len``,
696 ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
697 ``d.len``, ``d.flags``, ``d.id``
699 #. Set ``free_head`` to ``desc[free_head].next``
701 #. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``
703 When supplying used buffers to the driver:
705 1. Get corresponding used head-descriptor entry from descriptor ring,
708 2. Get corresponding ``DescStatePacked`` entry, ``e``
710 3. Set ``desc[e.last].next`` to ``free_head``
712 4. Set ``free_head`` to the index of ``e``
714 #. Steps 1,2,3,4 may be performed repeatedly if batching is possible
716 #. Increase ``used_idx`` by the size of the batch and update
717 ``used_wrap_counter`` if needed
719 #. Update ``d.flags``
721 #. Set the ``inflight`` field of each head ``DescStatePacked`` entry
724 #. Set ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
725 to ``free_head``, ``used_idx``, ``used_wrap_counter``
729 #. If ``used_idx`` does not match ``old_used_idx`` (means the
730 ``inflight`` field of ``DescStatePacked`` entries in last batch may
733 a. Get the next descriptor ring entry through ``old_used_idx``, ``d``
735 #. Use ``old_used_wrap_counter`` to calculate the available flags
737 #. If ``d.flags`` is not equal to the calculated flags value (means
738 slave has submitted the buffer to guest driver before crash, so
739 it has to commit the in-progres update), set ``old_free_head``,
740 ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
741 ``used_idx``, ``used_wrap_counter``
743 #. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
744 ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
745 (roll back any in-progress update)
747 #. Set the ``inflight`` field of each ``DescStatePacked`` entry in
750 #. Resubmit inflight ``DescStatePacked`` entries in order of their
758 #define VHOST_USER_PROTOCOL_F_MQ 0
759 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
760 #define VHOST_USER_PROTOCOL_F_RARP 2
761 #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
762 #define VHOST_USER_PROTOCOL_F_MTU 4
763 #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
764 #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
765 #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
766 #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
767 #define VHOST_USER_PROTOCOL_F_CONFIG 9
768 #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
769 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
770 #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
775 ``VHOST_USER_GET_FEATURES``
777 :equivalent ioctl: ``VHOST_GET_FEATURES``
779 :slave payload: ``u64``
781 Get from the underlying vhost implementation the features bitmask.
782 Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals slave support
783 for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
784 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
786 ``VHOST_USER_SET_FEATURES``
788 :equivalent ioctl: ``VHOST_SET_FEATURES``
789 :master payload: ``u64``
791 Enable features in the underlying vhost implementation using a
792 bitmask. Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
793 slave support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
794 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
796 ``VHOST_USER_GET_PROTOCOL_FEATURES``
798 :equivalent ioctl: ``VHOST_GET_FEATURES``
800 :slave payload: ``u64``
802 Get the protocol feature bitmask from the underlying vhost
803 implementation. Only legal if feature bit
804 ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
805 ``VHOST_USER_GET_FEATURES``.
808 Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must
809 support this message even before ``VHOST_USER_SET_FEATURES`` was
812 ``VHOST_USER_SET_PROTOCOL_FEATURES``
814 :equivalent ioctl: ``VHOST_SET_FEATURES``
815 :master payload: ``u64``
817 Enable protocol features in the underlying vhost implementation.
819 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
820 ``VHOST_USER_GET_FEATURES``.
823 Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
824 this message even before ``VHOST_USER_SET_FEATURES`` was called.
826 ``VHOST_USER_SET_OWNER``
828 :equivalent ioctl: ``VHOST_SET_OWNER``
831 Issued when a new connection is established. It sets the current
832 *master* as an owner of the session. This can be used on the *slave*
833 as a "session start" flag.
835 ``VHOST_USER_RESET_OWNER``
839 .. admonition:: Deprecated
841 This is no longer used. Used to be sent to request disabling all
842 rings, but some clients interpreted it to also discard connection
843 state (this interpretation would lead to bugs). It is recommended
844 that clients either ignore this message, or use it to disable all
847 ``VHOST_USER_SET_MEM_TABLE``
849 :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
850 :master payload: memory regions description
851 :slave payload: (postcopy only) memory regions description
853 Sets the memory map regions on the slave so it can translate the
854 vring addresses. In the ancillary data there is an array of file
855 descriptors for each memory mapped region. The size and ordering of
856 the fds matches the number and ordering of memory regions.
858 When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
859 ``SET_MEM_TABLE`` replies with the bases of the memory mapped
860 regions to the master. The slave must have mmap'd the regions but
861 not yet accessed them and should not yet generate a userfault
865 ``NEED_REPLY_MASK`` is not set in this case. QEMU will then
866 reply back to the list of mappings with an empty
867 ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
868 reception of this message may the guest start accessing the memory
869 and generating faults.
871 ``VHOST_USER_SET_LOG_BASE``
873 :equivalent ioctl: ``VHOST_SET_LOG_BASE``
877 Sets logging shared memory space.
879 When slave has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
880 the log memory fd is provided in the ancillary data of
881 ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
882 memory area provided in the message.
884 ``VHOST_USER_SET_LOG_FD``
886 :equivalent ioctl: ``VHOST_SET_LOG_FD``
889 Sets the logging file descriptor, which is passed as ancillary data.
891 ``VHOST_USER_SET_VRING_NUM``
893 :equivalent ioctl: ``VHOST_SET_VRING_NUM``
894 :master payload: vring state description
896 Set the size of the queue.
898 ``VHOST_USER_SET_VRING_ADDR``
900 :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
901 :master payload: vring address description
904 Sets the addresses of the different aspects of the vring.
906 ``VHOST_USER_SET_VRING_BASE``
908 :equivalent ioctl: ``VHOST_SET_VRING_BASE``
909 :master payload: vring state description
911 Sets the base offset in the available vring.
913 ``VHOST_USER_GET_VRING_BASE``
915 :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
916 :master payload: vring state description
917 :slave payload: vring state description
919 Get the available vring base offset.
921 ``VHOST_USER_SET_VRING_KICK``
923 :equivalent ioctl: ``VHOST_SET_VRING_KICK``
924 :master payload: ``u64``
926 Set the event file descriptor for adding buffers to the vring. It is
927 passed in the ancillary data.
929 Bits (0-7) of the payload contain the vring index. Bit 8 is the
930 invalid FD flag. This flag is set when there is no file descriptor
931 in the ancillary data. This signals that polling should be used
932 instead of waiting for a kick.
934 ``VHOST_USER_SET_VRING_CALL``
936 :equivalent ioctl: ``VHOST_SET_VRING_CALL``
937 :master payload: ``u64``
939 Set the event file descriptor to signal when buffers are used. It is
940 passed in the ancillary data.
942 Bits (0-7) of the payload contain the vring index. Bit 8 is the
943 invalid FD flag. This flag is set when there is no file descriptor
944 in the ancillary data. This signals that polling will be used
945 instead of waiting for the call.
947 ``VHOST_USER_SET_VRING_ERR``
949 :equivalent ioctl: ``VHOST_SET_VRING_ERR``
950 :master payload: ``u64``
952 Set the event file descriptor to signal when error occurs. It is
953 passed in the ancillary data.
955 Bits (0-7) of the payload contain the vring index. Bit 8 is the
956 invalid FD flag. This flag is set when there is no file descriptor
957 in the ancillary data.
959 ``VHOST_USER_GET_QUEUE_NUM``
961 :equivalent ioctl: N/A
965 Query how many queues the backend supports.
967 This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
968 is set in queried protocol features by
969 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
971 ``VHOST_USER_SET_VRING_ENABLE``
973 :equivalent ioctl: N/A
974 :master payload: vring state description
976 Signal slave to enable or disable corresponding vring.
978 This request should be sent only when
979 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.
981 ``VHOST_USER_SEND_RARP``
983 :equivalent ioctl: N/A
984 :master payload: ``u64``
986 Ask vhost user backend to broadcast a fake RARP to notify the migration
987 is terminated for guest that does not support GUEST_ANNOUNCE.
989 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
990 present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
991 ``VHOST_USER_PROTOCOL_F_RARP`` is present in
992 ``VHOST_USER_GET_PROTOCOL_FEATURES``. The first 6 bytes of the
993 payload contain the mac address of the guest to allow the vhost user
994 backend to construct and broadcast the fake RARP.
996 ``VHOST_USER_NET_SET_MTU``
998 :equivalent ioctl: N/A
999 :master payload: ``u64``
1001 Set host MTU value exposed to the guest.
1003 This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
1004 has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
1005 is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
1006 ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
1007 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
1009 If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
1010 respond with zero in case the specified MTU is valid, or non-zero
1013 ``VHOST_USER_SET_SLAVE_REQ_FD``
1015 :equivalent ioctl: N/A
1016 :master payload: N/A
1018 Set the socket file descriptor for slave initiated requests. It is passed
1019 in the ancillary data.
1021 This request should be sent only when
1022 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
1023 feature bit ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` bit is present in
1024 ``VHOST_USER_GET_PROTOCOL_FEATURES``. If
1025 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
1026 respond with zero for success, non-zero otherwise.
1028 ``VHOST_USER_IOTLB_MSG``
1030 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1031 :master payload: ``struct vhost_iotlb_msg``
1032 :slave payload: ``u64``
1034 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1036 Master sends such requests to update and invalidate entries in the
1037 device IOTLB. The slave has to acknowledge the request with sending
1038 zero as ``u64`` payload for success, non-zero otherwise.
1040 This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
1041 feature has been successfully negotiated.
1043 ``VHOST_USER_SET_VRING_ENDIAN``
1045 :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
1046 :master payload: vring state description
1048 Set the endianness of a VQ for legacy devices. Little-endian is
1049 indicated with state.num set to 0 and big-endian is indicated with
1050 state.num set to 1. Other values are invalid.
1052 This request should be sent only when
1053 ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
1054 Backends that negotiated this feature should handle both
1055 endiannesses and expect this message once (per VQ) during device
1056 configuration (ie. before the master starts the VQ).
1058 ``VHOST_USER_GET_CONFIG``
1060 :equivalent ioctl: N/A
1061 :master payload: virtio device config space
1062 :slave payload: virtio device config space
1064 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1065 submitted by the vhost-user master to fetch the contents of the
1066 virtio device configuration space, vhost-user slave's payload size
1067 MUST match master's request, vhost-user slave uses zero length of
1068 payload to indicate an error to vhost-user master. The vhost-user
1069 master may cache the contents to avoid repeated
1070 ``VHOST_USER_GET_CONFIG`` calls.
1072 ``VHOST_USER_SET_CONFIG``
1074 :equivalent ioctl: N/A
1075 :master payload: virtio device config space
1078 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1079 submitted by the vhost-user master when the Guest changes the virtio
1080 device configuration space and also can be used for live migration
1081 on the destination host. The vhost-user slave must check the flags
1082 field, and slaves MUST NOT accept SET_CONFIG for read-only
1083 configuration space fields unless the live migration bit is set.
1085 ``VHOST_USER_CREATE_CRYPTO_SESSION``
1087 :equivalent ioctl: N/A
1088 :master payload: crypto session description
1089 :slave payload: crypto session description
1091 Create a session for crypto operation. The server side must return
1092 the session id, 0 or positive for success, negative for failure.
1093 This request should be sent only when
1094 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1095 successfully negotiated. It's a required feature for crypto
1098 ``VHOST_USER_CLOSE_CRYPTO_SESSION``
1100 :equivalent ioctl: N/A
1101 :master payload: ``u64``
1103 Close a session for crypto operation which was previously
1104 created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.
1106 This request should be sent only when
1107 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1108 successfully negotiated. It's a required feature for crypto
1111 ``VHOST_USER_POSTCOPY_ADVISE``
1113 :master payload: N/A
1114 :slave payload: userfault fd
1116 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the master
1117 advises slave that a migration with postcopy enabled is underway,
1118 the slave must open a userfaultfd for later use. Note that at this
1119 stage the migration is still in precopy mode.
1121 ``VHOST_USER_POSTCOPY_LISTEN``
1123 :master payload: N/A
1125 Master advises slave that a transition to postcopy mode has
1126 happened. The slave must ensure that shared memory is registered
1127 with userfaultfd to cause faulting of non-present pages.
1129 This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
1130 and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.
1132 ``VHOST_USER_POSTCOPY_END``
1134 :slave payload: ``u64``
1136 Master advises that postcopy migration has now completed. The slave
1137 must disable the userfaultfd. The response is an acknowledgement
1140 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
1141 is sent at the end of the migration, after
1142 ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.
1144 The value returned is an error indication; 0 is success.
1146 ``VHOST_USER_GET_INFLIGHT_FD``
1148 :equivalent ioctl: N/A
1149 :master payload: inflight description
1151 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1152 been successfully negotiated, this message is submitted by master to
1153 get a shared buffer from slave. The shared buffer will be used to
1154 track inflight I/O by slave. QEMU should retrieve a new one when vm
1157 ``VHOST_USER_SET_INFLIGHT_FD``
1159 :equivalent ioctl: N/A
1160 :master payload: inflight description
1162 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1163 been successfully negotiated, this message is submitted by master to
1164 send the shared inflight buffer back to slave so that slave could
1165 get inflight I/O after a crash or restart.
1167 ``VHOST_USER_GPU_SET_SOCKET``
1169 :equivalent ioctl: N/A
1170 :master payload: N/A
1172 Sets the GPU protocol socket file descriptor, which is passed as
1173 ancillary data. The GPU protocol is used to inform the master of
1174 rendering state and updates. See vhost-user-gpu.rst for details.
1179 ``VHOST_USER_SLAVE_IOTLB_MSG``
1181 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1182 :slave payload: ``struct vhost_iotlb_msg``
1183 :master payload: N/A
1185 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1186 Slave sends such requests to notify of an IOTLB miss, or an IOTLB
1187 access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
1188 negotiated, and slave set the ``VHOST_USER_NEED_REPLY`` flag, master
1189 must respond with zero when operation is successfully completed, or
1190 non-zero otherwise. This request should be send only when
1191 ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
1194 ``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``
1196 :equivalent ioctl: N/A
1198 :master payload: N/A
1200 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
1201 slave sends such messages to notify that the virtio device's
1202 configuration space has changed, for those host devices which can
1203 support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
1204 message to slave to get the latest content. If
1205 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and slave set the
1206 ``VHOST_USER_NEED_REPLY`` flag, master must respond with zero when
1207 operation is successfully completed, or non-zero otherwise.
1209 ``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``
1211 :equivalent ioctl: N/A
1212 :slave payload: vring area description
1213 :master payload: N/A
1215 Sets host notifier for a specified queue. The queue index is
1216 contained in the ``u64`` field of the vring area description. The
1217 host notifier is described by the file descriptor (typically it's a
1218 VFIO device fd) which is passed as ancillary data and the size
1219 (which is mmap size and should be the same as host page size) and
1220 offset (which is mmap offset) carried in the vring area
1221 description. QEMU can mmap the file descriptor based on the size and
1222 offset to get a memory range. Registering a host notifier means
1223 mapping this memory range to the VM as the specified queue's notify
1224 MMIO region. Slave sends this request to tell QEMU to de-register
1225 the existing notifier if any and register the new notifier if the
1226 request is sent with a file descriptor.
1228 This request should be sent only when
1229 ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
1230 successfully negotiated.
1234 VHOST_USER_PROTOCOL_F_REPLY_ACK
1235 -------------------------------
1237 The original vhost-user specification only demands replies for certain
1238 commands. This differs from the vhost protocol implementation where
1239 commands are sent over an ``ioctl()`` call and block until the client
1242 With this protocol extension negotiated, the sender (QEMU) can set the
1243 ``need_reply`` [Bit 3] flag to any command. This indicates that the
1244 client MUST respond with a Payload ``VhostUserMsg`` indicating success
1245 or failure. The payload should be set to zero on success or non-zero
1246 on failure, unless the message already has an explicit reply body.
1248 The response payload gives QEMU a deterministic indication of the result
1249 of the command. Today, QEMU is expected to terminate the main vhost-user
1250 loop upon receiving such errors. In future, qemu could be taught to be more
1251 resilient for selective requests.
1253 For the message types that already solicit a reply from the client,
1254 the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
1255 being set brings no behavioural change. (See the Communication_
1256 section for details.)
1258 .. _backend_conventions:
1260 Backend program conventions
1261 ===========================
1263 vhost-user backends can provide various devices & services and may
1264 need to be configured manually depending on the use case. However, it
1265 is a good idea to follow the conventions listed here when
1266 possible. Users, QEMU or libvirt, can then rely on some common
1267 behaviour to avoid heterogenous configuration and management of the
1268 backend programs and facilitate interoperability.
1270 Each backend installed on a host system should come with at least one
1271 JSON file that conforms to the vhost-user.json schema. Each file
1272 informs the management applications about the backend type, and binary
1273 location. In addition, it defines rules for management apps for
1274 picking the highest priority backend when multiple match the search
1275 criteria (see ``@VhostUserBackend`` documentation in the schema file).
1277 If the backend is not capable of enabling a requested feature on the
1278 host (such as 3D acceleration with virgl), or the initialization
1279 failed, the backend should fail to start early and exit with a status
1280 != 0. It may also print a message to stderr for further details.
1282 The backend program must not daemonize itself, but it may be
1283 daemonized by the management layer. It may also have a restricted
1284 access to the system.
1286 File descriptors 0, 1 and 2 will exist, and have regular
1287 stdin/stdout/stderr usage (they may have been redirected to /dev/null
1288 by the management layer, or to a log handler).
1290 The backend program must end (as quickly and cleanly as possible) when
1291 the SIGTERM signal is received. Eventually, it may receive SIGKILL by
1292 the management layer after a few seconds.
1294 The following command line options have an expected behaviour. They
1295 are mandatory, unless explicitly said differently:
1299 This option specify the location of the vhost-user Unix domain socket.
1300 It is incompatible with --fd.
1304 When this argument is given, the backend program is started with the
1305 vhost-user socket as file descriptor FDNUM. It is incompatible with
1308 --print-capabilities
1310 Output to stdout the backend capabilities in JSON format, and then
1311 exit successfully. Other options and arguments should be ignored, and
1312 the backend program should not perform its normal function. The
1313 capabilities can be reported dynamically depending on the host
1316 The JSON output is described in the ``vhost-user.json`` schema, by
1317 ```@VHostUserBackendCapabilities``. Example:
1332 Command line options:
1336 Specify the linux input device.
1342 Do no request exclusive access to the input device.
1349 Command line options:
1353 Specify the GPU DRM render node.
1359 Enable virgl rendering support.