8 Copyright 2014 Virtual Open Systems Sarl.
9 Copyright 2019 Intel Corporation
10 Licence: This work is licensed under the terms of the GNU GPL,
11 version 2 or later. See the COPYING file in the top-level
14 .. contents:: Table of Contents
19 This protocol is aiming to complement the ``ioctl`` interface used to
20 control the vhost implementation in the Linux kernel. It implements
21 the control plane needed to establish virtqueue sharing with a user
22 space process on the same host. It uses communication over a Unix
23 domain socket to share file descriptors in the ancillary data of the
26 The protocol defines 2 sides of the communication, *master* and
27 *slave*. *Master* is the application that shares its virtqueues, in
28 our case QEMU. *Slave* is the consumer of the virtqueues.
30 In the current implementation QEMU is the *master*, and the *slave* is
31 the external process consuming the virtio queues, for example a
32 software Ethernet switch running in user space, such as Snabbswitch,
33 or a block device backend processing read & write to a virtual
34 disk. In order to facilitate interoperability between various backend
35 implementations, it is recommended to follow the :ref:`Backend program
36 conventions <backend_conventions>`.
38 *Master* and *slave* can be either a client (i.e. connecting) or
39 server (listening) in the socket communication.
41 Support for platforms other than Linux
42 --------------------------------------
44 While vhost-user was initially developed targeting Linux, nowadays it
45 is supported on any platform that provides the following features:
47 - A way for requesting shared memory represented by a file descriptor
48 so it can be passed over a UNIX domain socket and then mapped by the
51 - AF_UNIX sockets with SCM_RIGHTS, so QEMU and the other process can
52 exchange messages through it, including ancillary data when needed.
54 - Either eventfd or pipe/pipe2. On platforms where eventfd is not
55 available, QEMU will automatically fall back to pipe2 or, as a last
56 resort, pipe. Each file descriptor will be used for receiving or
57 sending events by reading or writing (respectively) an 8-byte value
58 to the corresponding it. The 8-value itself has no meaning and
59 should not be interpreted.
64 .. Note:: All numbers are in the machine native byte order.
66 A vhost-user message consists of 3 header fields and a payload.
68 +---------+-------+------+---------+
69 | request | flags | size | payload |
70 +---------+-------+------+---------+
75 :request: 32-bit type of the request
77 :flags: 32-bit bit field
79 - Lower 2 bits are the version (currently 0x01)
80 - Bit 2 is the reply flag - needs to be sent on each reply from the slave
81 - Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
84 :size: 32-bit size of the payload
89 Depending on the request type, **payload** can be:
91 A single 64-bit integer
92 ^^^^^^^^^^^^^^^^^^^^^^^
98 :u64: a 64-bit unsigned integer
100 A vring state description
101 ^^^^^^^^^^^^^^^^^^^^^^^^^
107 :index: a 32-bit index
109 :num: a 32-bit number
111 A vring address description
112 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
114 +-------+-------+------+------------+------+-----------+-----+
115 | index | flags | size | descriptor | used | available | log |
116 +-------+-------+------+------------+------+-----------+-----+
118 :index: a 32-bit vring index
120 :flags: a 32-bit vring flags
122 :descriptor: a 64-bit ring address of the vring descriptor table
124 :used: a 64-bit ring address of the vring used ring
126 :available: a 64-bit ring address of the vring available ring
128 :log: a 64-bit guest address for logging
130 Note that a ring address is an IOVA if ``VIRTIO_F_IOMMU_PLATFORM`` has
131 been negotiated. Otherwise it is a user address.
133 Memory regions description
134 ^^^^^^^^^^^^^^^^^^^^^^^^^^
136 +-------------+---------+---------+-----+---------+
137 | num regions | padding | region0 | ... | region7 |
138 +-------------+---------+---------+-----+---------+
140 :num regions: a 32-bit number of regions
146 +---------------+------+--------------+-------------+
147 | guest address | size | user address | mmap offset |
148 +---------------+------+--------------+-------------+
150 :guest address: a 64-bit guest address of the region
154 :user address: a 64-bit user address
156 :mmap offset: 64-bit offset where region starts in the mapped memory
158 Single memory region description
159 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
161 +---------+---------------+------+--------------+-------------+
162 | padding | guest address | size | user address | mmap offset |
163 +---------+---------------+------+--------------+-------------+
167 :guest address: a 64-bit guest address of the region
171 :user address: a 64-bit user address
173 :mmap offset: 64-bit offset where region starts in the mapped memory
178 +----------+------------+
179 | log size | log offset |
180 +----------+------------+
182 :log size: size of area used for logging
184 :log offset: offset from start of supplied file descriptor where
185 logging starts (i.e. where guest address 0 would be
191 +------+------+--------------+-------------------+------+
192 | iova | size | user address | permissions flags | type |
193 +------+------+--------------+-------------------+------+
195 :iova: a 64-bit I/O virtual address programmed by the guest
199 :user address: a 64-bit user address
201 :permissions flags: an 8-bit value:
205 - 3: Read/Write access
207 :type: an 8-bit IOTLB message type:
210 - 3: IOTLB invalidate
211 - 4: IOTLB access fail
213 Virtio device config space
214 ^^^^^^^^^^^^^^^^^^^^^^^^^^
216 +--------+------+-------+---------+
217 | offset | size | flags | payload |
218 +--------+------+-------+---------+
220 :offset: a 32-bit offset of virtio device's configuration space
222 :size: a 32-bit configuration space access size in bytes
224 :flags: a 32-bit value:
225 - 0: Vhost master messages used for writeable fields
226 - 1: Vhost master messages used for live migration
228 :payload: Size bytes array holding the contents of the virtio
229 device's configuration space
231 Vring area description
232 ^^^^^^^^^^^^^^^^^^^^^^
234 +-----+------+--------+
235 | u64 | size | offset |
236 +-----+------+--------+
238 :u64: a 64-bit integer contains vring index and flags
240 :size: a 64-bit size of this area
242 :offset: a 64-bit offset of this area from the start of the
243 supplied file descriptor
248 +-----------+-------------+------------+------------+
249 | mmap size | mmap offset | num queues | queue size |
250 +-----------+-------------+------------+------------+
252 :mmap size: a 64-bit size of area to track inflight I/O
254 :mmap offset: a 64-bit offset of this area from the start
255 of the supplied file descriptor
257 :num queues: a 16-bit number of virtqueues
259 :queue size: a 16-bit size of virtqueues
264 In QEMU the vhost-user message is implemented with the following struct:
268 typedef struct VhostUserMsg {
269 VhostUserRequest request;
274 struct vhost_vring_state state;
275 struct vhost_vring_addr addr;
276 VhostUserMemory memory;
278 struct vhost_iotlb_msg iotlb;
279 VhostUserConfig config;
280 VhostUserVringArea area;
281 VhostUserInflight inflight;
283 } QEMU_PACKED VhostUserMsg;
288 The protocol for vhost-user is based on the existing implementation of
289 vhost for the Linux Kernel. Most messages that can be sent via the
290 Unix domain socket implementing vhost-user have an equivalent ioctl to
291 the kernel implementation.
293 The communication consists of *master* sending message requests and
294 *slave* sending message replies. Most of the requests don't require
295 replies. Here is a list of the ones that do:
297 * ``VHOST_USER_GET_FEATURES``
298 * ``VHOST_USER_GET_PROTOCOL_FEATURES``
299 * ``VHOST_USER_GET_VRING_BASE``
300 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
301 * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
305 :ref:`REPLY_ACK <reply_ack>`
306 The section on ``REPLY_ACK`` protocol extension.
308 There are several messages that the master sends with file descriptors passed
309 in the ancillary data:
311 * ``VHOST_USER_ADD_MEM_REG``
312 * ``VHOST_USER_SET_MEM_TABLE``
313 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
314 * ``VHOST_USER_SET_LOG_FD``
315 * ``VHOST_USER_SET_VRING_KICK``
316 * ``VHOST_USER_SET_VRING_CALL``
317 * ``VHOST_USER_SET_VRING_ERR``
318 * ``VHOST_USER_SET_SLAVE_REQ_FD``
319 * ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
321 If *master* is unable to send the full message or receives a wrong
322 reply it will close the connection. An optional reconnection mechanism
325 If *slave* detects some error such as incompatible features, it may also
326 close the connection. This should only happen in exceptional circumstances.
328 Any protocol extensions are gated by protocol feature bits, which
329 allows full backwards compatibility on both master and slave. As
330 older slaves don't support negotiating protocol features, a feature
331 bit was dedicated for this purpose::
333 #define VHOST_USER_F_PROTOCOL_FEATURES 30
335 Starting and stopping rings
336 ---------------------------
338 Client must only process each ring when it is started.
340 Client must only pass data between the ring and the backend, when the
343 If ring is started but disabled, client must process the ring without
344 talking to the backend.
346 For example, for a networking device, in the disabled state client
347 must not supply any new RX packets, but must process and discard any
350 If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the
351 ring is initialized in an enabled state.
353 If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is
354 initialized in a disabled state. Client must not pass data to/from the
355 backend until ring is enabled by ``VHOST_USER_SET_VRING_ENABLE`` with
356 parameter 1, or after it has been disabled by
357 ``VHOST_USER_SET_VRING_ENABLE`` with parameter 0.
359 Each ring is initialized in a stopped state, client must not process
360 it until ring is started, or after it has been stopped.
362 Client must start ring upon receiving a kick (that is, detecting that
363 file descriptor is readable) on the descriptor specified by
364 ``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
365 ``VHOST_USER_VRING_KICK`` if negotiated, and stop ring upon receiving
366 ``VHOST_USER_GET_VRING_BASE``.
368 While processing the rings (whether they are enabled or not), client
369 must support changing some configuration aspects on the fly.
371 Multiple queue support
372 ----------------------
374 Many devices have a fixed number of virtqueues. In this case the master
375 already knows the number of available virtqueues without communicating with the
378 Some devices do not have a fixed number of virtqueues. Instead the maximum
379 number of virtqueues is chosen by the slave. The number can depend on host
380 resource availability or slave implementation details. Such devices are called
381 multiple queue devices.
383 Multiple queue support allows the slave to advertise the maximum number of
384 queues. This is treated as a protocol extension, hence the slave has to
385 implement protocol features first. The multiple queues feature is supported
386 only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ`` (bit 0) is set.
388 The max number of queues the slave supports can be queried with message
389 ``VHOST_USER_GET_QUEUE_NUM``. Master should stop when the number of requested
390 queues is bigger than that.
392 As all queues share one connection, the master uses a unique index for each
393 queue in the sent message to identify a specified queue.
395 The master enables queues by sending message ``VHOST_USER_SET_VRING_ENABLE``.
396 vhost-user-net has historically automatically enabled the first queue pair.
398 Slaves should always implement the ``VHOST_USER_PROTOCOL_F_MQ`` protocol
399 feature, even for devices with a fixed number of virtqueues, since it is simple
400 to implement and offers a degree of introspection.
402 Masters must not rely on the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature for
403 devices with a fixed number of virtqueues. Only true multiqueue devices
404 require this protocol feature.
409 During live migration, the master may need to track the modifications
410 the slave makes to the memory mapped regions. The client should mark
411 the dirty pages in a log. Once it complies to this logging, it may
412 declare the ``VHOST_F_LOG_ALL`` vhost feature.
414 To start/stop logging of data/used ring writes, server may send
415 messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
416 ``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
417 flags set to 1/0, respectively.
419 All the modifications to memory pointed by vring "descriptor" should
420 be marked. Modifications to "used" vring should be marked if
421 ``VHOST_VRING_F_LOG`` is part of ring's flags.
423 Dirty pages are of size::
425 #define VHOST_LOG_PAGE 0x1000
427 The log memory fd is provided in the ancillary data of
428 ``VHOST_USER_SET_LOG_BASE`` message when the slave has
429 ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.
431 The size of the log is supplied as part of ``VhostUserMsg`` which
432 should be large enough to cover all known guest addresses. Log starts
433 at the supplied offset in the supplied file descriptor. The log
434 covers from address 0 to the maximum of guest regions. In pseudo-code,
435 to mark page at ``addr`` as dirty::
437 page = addr / VHOST_LOG_PAGE
438 log[page / 8] |= 1 << page % 8
440 Where ``addr`` is the guest physical address.
442 Use atomic operations, as the log may be concurrently manipulated.
444 Note that when logging modifications to the used ring (when
445 ``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
446 be used to calculate the log offset: the write to first byte of the
447 used ring is logged at this offset from log start. Also note that this
448 value might be outside the legal guest physical address range
449 (i.e. does not have to be covered by the ``VhostUserMemory`` table), but
450 the bit offset of the last byte of the ring must fall within the size
451 supplied by ``VhostUserLog``.
453 ``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
454 ancillary data, it may be used to inform the master that the log has
457 Once the source has finished migration, rings will be stopped by the
458 source. No further update must be done before rings are restarted.
460 In postcopy migration the slave is started before all the memory has
461 been received from the source host, and care must be taken to avoid
462 accessing pages that have yet to be received. The slave opens a
463 'userfault'-fd and registers the memory with it; this fd is then
464 passed back over to the master. The master services requests on the
465 userfaultfd for pages that are accessed and when the page is available
466 it performs WAKE ioctl's on the userfaultfd to wake the stalled
467 slave. The client indicates support for this via the
468 ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.
473 The master sends a list of vhost memory regions to the slave using the
474 ``VHOST_USER_SET_MEM_TABLE`` message. Each region has two base
475 addresses: a guest address and a user address.
477 Messages contain guest addresses and/or user addresses to reference locations
478 within the shared memory. The mapping of these addresses works as follows.
480 User addresses map to the vhost memory region containing that user address.
482 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:
484 * Guest addresses map to the vhost memory region containing that guest
487 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:
489 * Guest addresses are also called I/O virtual addresses (IOVAs). They are
490 translated to user addresses via the IOTLB.
492 * The vhost memory region guest address is not used.
497 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
498 master sends IOTLB entries update & invalidation by sending
499 ``VHOST_USER_IOTLB_MSG`` requests to the slave with a ``struct
500 vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
501 has to be filled with the update message type (2), the I/O virtual
502 address, the size, the user virtual address, and the permissions
503 flags. Addresses and size must be within vhost memory regions set via
504 the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
505 ``iotlb`` payload has to be filled with the invalidation message type
506 (3), the I/O virtual address and the size. On success, the slave is
507 expected to reply with a zero payload, non-zero otherwise.
509 The slave relies on the slave communication channel (see :ref:`Slave
510 communication <slave_communication>` section below) to send IOTLB miss
511 and access failure events, by sending ``VHOST_USER_SLAVE_IOTLB_MSG``
512 requests to the master with a ``struct vhost_iotlb_msg`` as
513 payload. For miss events, the iotlb payload has to be filled with the
514 miss message type (1), the I/O virtual address and the permissions
515 flags. For access failure event, the iotlb payload has to be filled
516 with the access failure message type (4), the I/O virtual address and
517 the permissions flags. For synchronization purpose, the slave may
518 rely on the reply-ack feature, so the master may send a reply when
519 operation is completed if the reply-ack feature is negotiated and
520 slaves requests a reply. For miss events, completed operation means
521 either master sent an update message containing the IOTLB entry
522 containing requested address and permission, or master sent nothing if
523 the IOTLB miss message is invalid (invalid IOVA or permission).
525 The master isn't expected to take the initiative to send IOTLB update
526 messages, as the slave sends IOTLB miss messages for the guest virtual
527 memory areas it needs to access.
529 .. _slave_communication:
534 An optional communication channel is provided if the slave declares
535 ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` protocol feature, to allow the
536 slave to make requests to the master.
538 The fd is provided via ``VHOST_USER_SET_SLAVE_REQ_FD`` ancillary data.
540 A slave may then send ``VHOST_USER_SLAVE_*`` messages to the master
541 using this fd communication channel.
543 If ``VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD`` protocol feature is
544 negotiated, slave can send file descriptors (at most 8 descriptors in
545 each message) to master via ancillary data using this fd communication
548 Inflight I/O tracking
549 ---------------------
551 To support reconnecting after restart or crash, slave may need to
552 resubmit inflight I/Os. If virtqueue is processed in order, we can
553 easily achieve that by getting the inflight descriptors from
554 descriptor table (split virtqueue) or descriptor ring (packed
555 virtqueue). However, it can't work when we process descriptors
556 out-of-order because some entries which store the information of
557 inflight descriptors in available ring (split virtqueue) or descriptor
558 ring (packed virtqueue) might be overridden by new entries. To solve
559 this problem, slave need to allocate an extra buffer to store this
560 information of inflight descriptors and share it with master for
561 persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
562 ``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
563 between master and slave. And the format of this buffer is described
566 +---------------+---------------+-----+---------------+
567 | queue0 region | queue1 region | ... | queueN region |
568 +---------------+---------------+-----+---------------+
570 N is the number of available virtqueues. Slave could get it from num
571 queues field of ``VhostUserInflight``.
573 For split virtqueue, queue region can be implemented as:
577 typedef struct DescStateSplit {
578 /* Indicate whether this descriptor is inflight or not.
579 * Only available for head-descriptor. */
585 /* Maintain a list for the last batch of used descriptors.
586 * Only available when batching is used for submitting */
589 /* Used to preserve the order of fetching available descriptors.
590 * Only available for head-descriptor. */
594 typedef struct QueueRegionSplit {
595 /* The feature flags of this region. Now it's initialized to 0. */
598 /* The version of this region. It's 1 currently.
599 * Zero value indicates an uninitialized buffer */
602 /* The size of DescStateSplit array. It's equal to the virtqueue
603 * size. Slave could get it from queue size field of VhostUserInflight. */
606 /* The head of list that track the last batch of used descriptors. */
607 uint16_t last_batch_head;
609 /* Store the idx value of used ring */
612 /* Used to track the state of each descriptor in descriptor table */
613 DescStateSplit desc[];
616 To track inflight I/O, the queue region should be processed as follows:
618 When receiving available buffers from the driver:
620 #. Get the next available head-descriptor index from available ring, ``i``
622 #. Set ``desc[i].counter`` to the value of global counter
624 #. Increase global counter by 1
626 #. Set ``desc[i].inflight`` to 1
628 When supplying used buffers to the driver:
630 1. Get corresponding used head-descriptor index, i
632 2. Set ``desc[i].next`` to ``last_batch_head``
634 3. Set ``last_batch_head`` to ``i``
636 #. Steps 1,2,3 may be performed repeatedly if batching is possible
638 #. Increase the ``idx`` value of used ring by the size of the batch
640 #. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0
642 #. Set ``used_idx`` to the ``idx`` value of used ring
646 #. If the value of ``used_idx`` does not match the ``idx`` value of
647 used ring (means the inflight field of ``DescStateSplit`` entries in
648 last batch may be incorrect),
650 a. Subtract the value of ``used_idx`` from the ``idx`` value of
651 used ring to get last batch size of ``DescStateSplit`` entries
653 #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
654 list which starts from ``last_batch_head``
656 #. Set ``used_idx`` to the ``idx`` value of used ring
658 #. Resubmit inflight ``DescStateSplit`` entries in order of their
661 For packed virtqueue, queue region can be implemented as:
665 typedef struct DescStatePacked {
666 /* Indicate whether this descriptor is inflight or not.
667 * Only available for head-descriptor. */
673 /* Link to the next free entry */
676 /* Link to the last entry of descriptor list.
677 * Only available for head-descriptor. */
680 /* The length of descriptor list.
681 * Only available for head-descriptor. */
684 /* Used to preserve the order of fetching available descriptors.
685 * Only available for head-descriptor. */
691 /* The descriptor flags */
694 /* The buffer length */
697 /* The buffer address */
701 typedef struct QueueRegionPacked {
702 /* The feature flags of this region. Now it's initialized to 0. */
705 /* The version of this region. It's 1 currently.
706 * Zero value indicates an uninitialized buffer */
709 /* The size of DescStatePacked array. It's equal to the virtqueue
710 * size. Slave could get it from queue size field of VhostUserInflight. */
713 /* The head of free DescStatePacked entry list */
716 /* The old head of free DescStatePacked entry list */
717 uint16_t old_free_head;
719 /* The used index of descriptor ring */
722 /* The old used index of descriptor ring */
723 uint16_t old_used_idx;
725 /* Device ring wrap counter */
726 uint8_t used_wrap_counter;
728 /* The old device ring wrap counter */
729 uint8_t old_used_wrap_counter;
734 /* Used to track the state of each descriptor fetched from descriptor ring */
735 DescStatePacked desc[];
738 To track inflight I/O, the queue region should be processed as follows:
740 When receiving available buffers from the driver:
742 #. Get the next available descriptor entry from descriptor ring, ``d``
744 #. If ``d`` is head descriptor,
746 a. Set ``desc[old_free_head].num`` to 0
748 #. Set ``desc[old_free_head].counter`` to the value of global counter
750 #. Increase global counter by 1
752 #. Set ``desc[old_free_head].inflight`` to 1
754 #. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
757 #. Increase ``desc[old_free_head].num`` by 1
759 #. Set ``desc[free_head].addr``, ``desc[free_head].len``,
760 ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
761 ``d.len``, ``d.flags``, ``d.id``
763 #. Set ``free_head`` to ``desc[free_head].next``
765 #. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``
767 When supplying used buffers to the driver:
769 1. Get corresponding used head-descriptor entry from descriptor ring,
772 2. Get corresponding ``DescStatePacked`` entry, ``e``
774 3. Set ``desc[e.last].next`` to ``free_head``
776 4. Set ``free_head`` to the index of ``e``
778 #. Steps 1,2,3,4 may be performed repeatedly if batching is possible
780 #. Increase ``used_idx`` by the size of the batch and update
781 ``used_wrap_counter`` if needed
783 #. Update ``d.flags``
785 #. Set the ``inflight`` field of each head ``DescStatePacked`` entry
788 #. Set ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
789 to ``free_head``, ``used_idx``, ``used_wrap_counter``
793 #. If ``used_idx`` does not match ``old_used_idx`` (means the
794 ``inflight`` field of ``DescStatePacked`` entries in last batch may
797 a. Get the next descriptor ring entry through ``old_used_idx``, ``d``
799 #. Use ``old_used_wrap_counter`` to calculate the available flags
801 #. If ``d.flags`` is not equal to the calculated flags value (means
802 slave has submitted the buffer to guest driver before crash, so
803 it has to commit the in-progres update), set ``old_free_head``,
804 ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
805 ``used_idx``, ``used_wrap_counter``
807 #. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
808 ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
809 (roll back any in-progress update)
811 #. Set the ``inflight`` field of each ``DescStatePacked`` entry in
814 #. Resubmit inflight ``DescStatePacked`` entries in order of their
817 In-band notifications
818 ---------------------
820 In some limited situations (e.g. for simulation) it is desirable to
821 have the kick, call and error (if used) signals done via in-band
822 messages instead of asynchronous eventfd notifications. This can be
823 done by negotiating the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS``
826 Note that due to the fact that too many messages on the sockets can
827 cause the sending application(s) to block, it is not advised to use
828 this feature unless absolutely necessary. It is also considered an
829 error to negotiate this feature without also negotiating
830 ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` and ``VHOST_USER_PROTOCOL_F_REPLY_ACK``,
831 the former is necessary for getting a message channel from the slave
832 to the master, while the latter needs to be used with the in-band
833 notification messages to block until they are processed, both to avoid
834 blocking later and for proper processing (at least in the simulation
835 use case.) As it has no other way of signalling this error, the slave
836 should close the connection as a response to a
837 ``VHOST_USER_SET_PROTOCOL_FEATURES`` message that sets the in-band
838 notifications feature flag without the other two.
845 #define VHOST_USER_PROTOCOL_F_MQ 0
846 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
847 #define VHOST_USER_PROTOCOL_F_RARP 2
848 #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
849 #define VHOST_USER_PROTOCOL_F_MTU 4
850 #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
851 #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
852 #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
853 #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
854 #define VHOST_USER_PROTOCOL_F_CONFIG 9
855 #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
856 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
857 #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
858 #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13
859 #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14
860 #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
861 #define VHOST_USER_PROTOCOL_F_STATUS 16
866 ``VHOST_USER_GET_FEATURES``
868 :equivalent ioctl: ``VHOST_GET_FEATURES``
870 :slave payload: ``u64``
872 Get from the underlying vhost implementation the features bitmask.
873 Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals slave support
874 for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
875 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
877 ``VHOST_USER_SET_FEATURES``
879 :equivalent ioctl: ``VHOST_SET_FEATURES``
880 :master payload: ``u64``
882 Enable features in the underlying vhost implementation using a
883 bitmask. Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
884 slave support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
885 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
887 ``VHOST_USER_GET_PROTOCOL_FEATURES``
889 :equivalent ioctl: ``VHOST_GET_FEATURES``
891 :slave payload: ``u64``
893 Get the protocol feature bitmask from the underlying vhost
894 implementation. Only legal if feature bit
895 ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
896 ``VHOST_USER_GET_FEATURES``.
899 Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must
900 support this message even before ``VHOST_USER_SET_FEATURES`` was
903 ``VHOST_USER_SET_PROTOCOL_FEATURES``
905 :equivalent ioctl: ``VHOST_SET_FEATURES``
906 :master payload: ``u64``
908 Enable protocol features in the underlying vhost implementation.
910 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
911 ``VHOST_USER_GET_FEATURES``.
914 Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
915 this message even before ``VHOST_USER_SET_FEATURES`` was called.
917 ``VHOST_USER_SET_OWNER``
919 :equivalent ioctl: ``VHOST_SET_OWNER``
922 Issued when a new connection is established. It sets the current
923 *master* as an owner of the session. This can be used on the *slave*
924 as a "session start" flag.
926 ``VHOST_USER_RESET_OWNER``
930 .. admonition:: Deprecated
932 This is no longer used. Used to be sent to request disabling all
933 rings, but some clients interpreted it to also discard connection
934 state (this interpretation would lead to bugs). It is recommended
935 that clients either ignore this message, or use it to disable all
938 ``VHOST_USER_SET_MEM_TABLE``
940 :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
941 :master payload: memory regions description
942 :slave payload: (postcopy only) memory regions description
944 Sets the memory map regions on the slave so it can translate the
945 vring addresses. In the ancillary data there is an array of file
946 descriptors for each memory mapped region. The size and ordering of
947 the fds matches the number and ordering of memory regions.
949 When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
950 ``SET_MEM_TABLE`` replies with the bases of the memory mapped
951 regions to the master. The slave must have mmap'd the regions but
952 not yet accessed them and should not yet generate a userfault
956 ``NEED_REPLY_MASK`` is not set in this case. QEMU will then
957 reply back to the list of mappings with an empty
958 ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
959 reception of this message may the guest start accessing the memory
960 and generating faults.
962 ``VHOST_USER_SET_LOG_BASE``
964 :equivalent ioctl: ``VHOST_SET_LOG_BASE``
968 Sets logging shared memory space.
970 When slave has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
971 the log memory fd is provided in the ancillary data of
972 ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
973 memory area provided in the message.
975 ``VHOST_USER_SET_LOG_FD``
977 :equivalent ioctl: ``VHOST_SET_LOG_FD``
980 Sets the logging file descriptor, which is passed as ancillary data.
982 ``VHOST_USER_SET_VRING_NUM``
984 :equivalent ioctl: ``VHOST_SET_VRING_NUM``
985 :master payload: vring state description
987 Set the size of the queue.
989 ``VHOST_USER_SET_VRING_ADDR``
991 :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
992 :master payload: vring address description
995 Sets the addresses of the different aspects of the vring.
997 ``VHOST_USER_SET_VRING_BASE``
999 :equivalent ioctl: ``VHOST_SET_VRING_BASE``
1000 :master payload: vring state description
1002 Sets the base offset in the available vring.
1004 ``VHOST_USER_GET_VRING_BASE``
1006 :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
1007 :master payload: vring state description
1008 :slave payload: vring state description
1010 Get the available vring base offset.
1012 ``VHOST_USER_SET_VRING_KICK``
1014 :equivalent ioctl: ``VHOST_SET_VRING_KICK``
1015 :master payload: ``u64``
1017 Set the event file descriptor for adding buffers to the vring. It is
1018 passed in the ancillary data.
1020 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1021 invalid FD flag. This flag is set when there is no file descriptor
1022 in the ancillary data. This signals that polling should be used
1023 instead of waiting for the kick. Note that if the protocol feature
1024 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` has been negotiated
1025 this message isn't necessary as the ring is also started on the
1026 ``VHOST_USER_VRING_KICK`` message, it may however still be used to
1027 set an event file descriptor (which will be preferred over the
1028 message) or to enable polling.
1030 ``VHOST_USER_SET_VRING_CALL``
1032 :equivalent ioctl: ``VHOST_SET_VRING_CALL``
1033 :master payload: ``u64``
1035 Set the event file descriptor to signal when buffers are used. It is
1036 passed in the ancillary data.
1038 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1039 invalid FD flag. This flag is set when there is no file descriptor
1040 in the ancillary data. This signals that polling will be used
1041 instead of waiting for the call. Note that if the protocol features
1042 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
1043 ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` have been negotiated this message
1044 isn't necessary as the ``VHOST_USER_SLAVE_VRING_CALL`` message can be
1045 used, it may however still be used to set an event file descriptor
1046 or to enable polling.
1048 ``VHOST_USER_SET_VRING_ERR``
1050 :equivalent ioctl: ``VHOST_SET_VRING_ERR``
1051 :master payload: ``u64``
1053 Set the event file descriptor to signal when error occurs. It is
1054 passed in the ancillary data.
1056 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1057 invalid FD flag. This flag is set when there is no file descriptor
1058 in the ancillary data. Note that if the protocol features
1059 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
1060 ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` have been negotiated this message
1061 isn't necessary as the ``VHOST_USER_SLAVE_VRING_ERR`` message can be
1062 used, it may however still be used to set an event file descriptor
1063 (which will be preferred over the message).
1065 ``VHOST_USER_GET_QUEUE_NUM``
1067 :equivalent ioctl: N/A
1068 :master payload: N/A
1071 Query how many queues the backend supports.
1073 This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
1074 is set in queried protocol features by
1075 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
1077 ``VHOST_USER_SET_VRING_ENABLE``
1079 :equivalent ioctl: N/A
1080 :master payload: vring state description
1082 Signal slave to enable or disable corresponding vring.
1084 This request should be sent only when
1085 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.
1087 ``VHOST_USER_SEND_RARP``
1089 :equivalent ioctl: N/A
1090 :master payload: ``u64``
1092 Ask vhost user backend to broadcast a fake RARP to notify the migration
1093 is terminated for guest that does not support GUEST_ANNOUNCE.
1095 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
1096 present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
1097 ``VHOST_USER_PROTOCOL_F_RARP`` is present in
1098 ``VHOST_USER_GET_PROTOCOL_FEATURES``. The first 6 bytes of the
1099 payload contain the mac address of the guest to allow the vhost user
1100 backend to construct and broadcast the fake RARP.
1102 ``VHOST_USER_NET_SET_MTU``
1104 :equivalent ioctl: N/A
1105 :master payload: ``u64``
1107 Set host MTU value exposed to the guest.
1109 This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
1110 has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
1111 is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
1112 ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
1113 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
1115 If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
1116 respond with zero in case the specified MTU is valid, or non-zero
1119 ``VHOST_USER_SET_SLAVE_REQ_FD``
1121 :equivalent ioctl: N/A
1122 :master payload: N/A
1124 Set the socket file descriptor for slave initiated requests. It is passed
1125 in the ancillary data.
1127 This request should be sent only when
1128 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
1129 feature bit ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` bit is present in
1130 ``VHOST_USER_GET_PROTOCOL_FEATURES``. If
1131 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
1132 respond with zero for success, non-zero otherwise.
1134 ``VHOST_USER_IOTLB_MSG``
1136 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1137 :master payload: ``struct vhost_iotlb_msg``
1138 :slave payload: ``u64``
1140 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1142 Master sends such requests to update and invalidate entries in the
1143 device IOTLB. The slave has to acknowledge the request with sending
1144 zero as ``u64`` payload for success, non-zero otherwise.
1146 This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
1147 feature has been successfully negotiated.
1149 ``VHOST_USER_SET_VRING_ENDIAN``
1151 :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
1152 :master payload: vring state description
1154 Set the endianness of a VQ for legacy devices. Little-endian is
1155 indicated with state.num set to 0 and big-endian is indicated with
1156 state.num set to 1. Other values are invalid.
1158 This request should be sent only when
1159 ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
1160 Backends that negotiated this feature should handle both
1161 endiannesses and expect this message once (per VQ) during device
1162 configuration (ie. before the master starts the VQ).
1164 ``VHOST_USER_GET_CONFIG``
1166 :equivalent ioctl: N/A
1167 :master payload: virtio device config space
1168 :slave payload: virtio device config space
1170 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1171 submitted by the vhost-user master to fetch the contents of the
1172 virtio device configuration space, vhost-user slave's payload size
1173 MUST match master's request, vhost-user slave uses zero length of
1174 payload to indicate an error to vhost-user master. The vhost-user
1175 master may cache the contents to avoid repeated
1176 ``VHOST_USER_GET_CONFIG`` calls.
1178 ``VHOST_USER_SET_CONFIG``
1180 :equivalent ioctl: N/A
1181 :master payload: virtio device config space
1184 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1185 submitted by the vhost-user master when the Guest changes the virtio
1186 device configuration space and also can be used for live migration
1187 on the destination host. The vhost-user slave must check the flags
1188 field, and slaves MUST NOT accept SET_CONFIG for read-only
1189 configuration space fields unless the live migration bit is set.
1191 ``VHOST_USER_CREATE_CRYPTO_SESSION``
1193 :equivalent ioctl: N/A
1194 :master payload: crypto session description
1195 :slave payload: crypto session description
1197 Create a session for crypto operation. The server side must return
1198 the session id, 0 or positive for success, negative for failure.
1199 This request should be sent only when
1200 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1201 successfully negotiated. It's a required feature for crypto
1204 ``VHOST_USER_CLOSE_CRYPTO_SESSION``
1206 :equivalent ioctl: N/A
1207 :master payload: ``u64``
1209 Close a session for crypto operation which was previously
1210 created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.
1212 This request should be sent only when
1213 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1214 successfully negotiated. It's a required feature for crypto
1217 ``VHOST_USER_POSTCOPY_ADVISE``
1219 :master payload: N/A
1220 :slave payload: userfault fd
1222 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the master
1223 advises slave that a migration with postcopy enabled is underway,
1224 the slave must open a userfaultfd for later use. Note that at this
1225 stage the migration is still in precopy mode.
1227 ``VHOST_USER_POSTCOPY_LISTEN``
1229 :master payload: N/A
1231 Master advises slave that a transition to postcopy mode has
1232 happened. The slave must ensure that shared memory is registered
1233 with userfaultfd to cause faulting of non-present pages.
1235 This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
1236 and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.
1238 ``VHOST_USER_POSTCOPY_END``
1240 :slave payload: ``u64``
1242 Master advises that postcopy migration has now completed. The slave
1243 must disable the userfaultfd. The response is an acknowledgement
1246 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
1247 is sent at the end of the migration, after
1248 ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.
1250 The value returned is an error indication; 0 is success.
1252 ``VHOST_USER_GET_INFLIGHT_FD``
1254 :equivalent ioctl: N/A
1255 :master payload: inflight description
1257 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1258 been successfully negotiated, this message is submitted by master to
1259 get a shared buffer from slave. The shared buffer will be used to
1260 track inflight I/O by slave. QEMU should retrieve a new one when vm
1263 ``VHOST_USER_SET_INFLIGHT_FD``
1265 :equivalent ioctl: N/A
1266 :master payload: inflight description
1268 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1269 been successfully negotiated, this message is submitted by master to
1270 send the shared inflight buffer back to slave so that slave could
1271 get inflight I/O after a crash or restart.
1273 ``VHOST_USER_GPU_SET_SOCKET``
1275 :equivalent ioctl: N/A
1276 :master payload: N/A
1278 Sets the GPU protocol socket file descriptor, which is passed as
1279 ancillary data. The GPU protocol is used to inform the master of
1280 rendering state and updates. See vhost-user-gpu.rst for details.
1282 ``VHOST_USER_RESET_DEVICE``
1284 :equivalent ioctl: N/A
1285 :master payload: N/A
1288 Ask the vhost user backend to disable all rings and reset all
1289 internal device state to the initial state, ready to be
1290 reinitialized. The backend retains ownership of the device
1291 throughout the reset operation.
1293 Only valid if the ``VHOST_USER_PROTOCOL_F_RESET_DEVICE`` protocol
1294 feature is set by the backend.
1296 ``VHOST_USER_VRING_KICK``
1298 :equivalent ioctl: N/A
1299 :slave payload: vring state description
1300 :master payload: N/A
1302 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1303 feature has been successfully negotiated, this message may be
1304 submitted by the master to indicate that a buffer was added to
1305 the vring instead of signalling it using the vring's kick file
1306 descriptor or having the slave rely on polling.
1308 The state.num field is currently reserved and must be set to 0.
1310 ``VHOST_USER_GET_MAX_MEM_SLOTS``
1312 :equivalent ioctl: N/A
1315 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1316 feature has been successfully negotiated, this message is submitted
1317 by master to the slave. The slave should return the message with a
1318 u64 payload containing the maximum number of memory slots for
1319 QEMU to expose to the guest. The value returned by the backend
1320 will be capped at the maximum number of ram slots which can be
1321 supported by the target platform.
1323 ``VHOST_USER_ADD_MEM_REG``
1325 :equivalent ioctl: N/A
1326 :slave payload: single memory region description
1328 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1329 feature has been successfully negotiated, this message is submitted
1330 by the master to the slave. The message payload contains a memory
1331 region descriptor struct, describing a region of guest memory which
1332 the slave device must map in. When the
1333 ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has
1334 been successfully negotiated, along with the
1335 ``VHOST_USER_REM_MEM_REG`` message, this message is used to set and
1336 update the memory tables of the slave device.
1338 Exactly one file descriptor from which the memory is mapped is
1339 passed in the ancillary data.
1341 In postcopy mode (see ``VHOST_USER_POSTCOPY_LISTEN``), the slave
1342 replies with the bases of the memory mapped region to the master.
1343 For further details on postcopy, see ``VHOST_USER_SET_MEM_TABLE``.
1344 They apply to ``VHOST_USER_ADD_MEM_REG`` accordingly.
1346 ``VHOST_USER_REM_MEM_REG``
1348 :equivalent ioctl: N/A
1349 :slave payload: single memory region description
1351 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1352 feature has been successfully negotiated, this message is submitted
1353 by the master to the slave. The message payload contains a memory
1354 region descriptor struct, describing a region of guest memory which
1355 the slave device must unmap. When the
1356 ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has
1357 been successfully negotiated, along with the
1358 ``VHOST_USER_ADD_MEM_REG`` message, this message is used to set and
1359 update the memory tables of the slave device.
1361 The memory region to be removed is identified by its guest address,
1362 user address and size. The mmap offset is ignored.
1364 No file descriptors SHOULD be passed in the ancillary data. For
1365 compatibility with existing incorrect implementations, the slave MAY
1366 accept messages with one file descriptor. If a file descriptor is
1367 passed, the slave MUST close it without using it otherwise.
1369 ``VHOST_USER_SET_STATUS``
1371 :equivalent ioctl: VHOST_VDPA_SET_STATUS
1373 :master payload: ``u64``
1375 When the ``VHOST_USER_PROTOCOL_F_STATUS`` protocol feature has been
1376 successfully negotiated, this message is submitted by the master to
1377 notify the backend with updated device status as defined in the Virtio
1380 ``VHOST_USER_GET_STATUS``
1382 :equivalent ioctl: VHOST_VDPA_GET_STATUS
1383 :slave payload: ``u64``
1384 :master payload: N/A
1386 When the ``VHOST_USER_PROTOCOL_F_STATUS`` protocol feature has been
1387 successfully negotiated, this message is submitted by the master to
1388 query the backend for its device status as defined in the Virtio
1395 ``VHOST_USER_SLAVE_IOTLB_MSG``
1397 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1398 :slave payload: ``struct vhost_iotlb_msg``
1399 :master payload: N/A
1401 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1402 Slave sends such requests to notify of an IOTLB miss, or an IOTLB
1403 access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
1404 negotiated, and slave set the ``VHOST_USER_NEED_REPLY`` flag, master
1405 must respond with zero when operation is successfully completed, or
1406 non-zero otherwise. This request should be send only when
1407 ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
1410 ``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``
1412 :equivalent ioctl: N/A
1414 :master payload: N/A
1416 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
1417 slave sends such messages to notify that the virtio device's
1418 configuration space has changed, for those host devices which can
1419 support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
1420 message to slave to get the latest content. If
1421 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and slave set the
1422 ``VHOST_USER_NEED_REPLY`` flag, master must respond with zero when
1423 operation is successfully completed, or non-zero otherwise.
1425 ``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``
1427 :equivalent ioctl: N/A
1428 :slave payload: vring area description
1429 :master payload: N/A
1431 Sets host notifier for a specified queue. The queue index is
1432 contained in the ``u64`` field of the vring area description. The
1433 host notifier is described by the file descriptor (typically it's a
1434 VFIO device fd) which is passed as ancillary data and the size
1435 (which is mmap size and should be the same as host page size) and
1436 offset (which is mmap offset) carried in the vring area
1437 description. QEMU can mmap the file descriptor based on the size and
1438 offset to get a memory range. Registering a host notifier means
1439 mapping this memory range to the VM as the specified queue's notify
1440 MMIO region. Slave sends this request to tell QEMU to de-register
1441 the existing notifier if any and register the new notifier if the
1442 request is sent with a file descriptor.
1444 This request should be sent only when
1445 ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
1446 successfully negotiated.
1448 ``VHOST_USER_SLAVE_VRING_CALL``
1450 :equivalent ioctl: N/A
1451 :slave payload: vring state description
1452 :master payload: N/A
1454 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1455 feature has been successfully negotiated, this message may be
1456 submitted by the slave to indicate that a buffer was used from
1457 the vring instead of signalling this using the vring's call file
1458 descriptor or having the master relying on polling.
1460 The state.num field is currently reserved and must be set to 0.
1462 ``VHOST_USER_SLAVE_VRING_ERR``
1464 :equivalent ioctl: N/A
1465 :slave payload: vring state description
1466 :master payload: N/A
1468 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1469 feature has been successfully negotiated, this message may be
1470 submitted by the slave to indicate that an error occurred on the
1471 specific vring, instead of signalling the error file descriptor
1472 set by the master via ``VHOST_USER_SET_VRING_ERR``.
1474 The state.num field is currently reserved and must be set to 0.
1478 VHOST_USER_PROTOCOL_F_REPLY_ACK
1479 -------------------------------
1481 The original vhost-user specification only demands replies for certain
1482 commands. This differs from the vhost protocol implementation where
1483 commands are sent over an ``ioctl()`` call and block until the client
1486 With this protocol extension negotiated, the sender (QEMU) can set the
1487 ``need_reply`` [Bit 3] flag to any command. This indicates that the
1488 client MUST respond with a Payload ``VhostUserMsg`` indicating success
1489 or failure. The payload should be set to zero on success or non-zero
1490 on failure, unless the message already has an explicit reply body.
1492 The response payload gives QEMU a deterministic indication of the result
1493 of the command. Today, QEMU is expected to terminate the main vhost-user
1494 loop upon receiving such errors. In future, qemu could be taught to be more
1495 resilient for selective requests.
1497 For the message types that already solicit a reply from the client,
1498 the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
1499 being set brings no behavioural change. (See the Communication_
1500 section for details.)
1502 .. _backend_conventions:
1504 Backend program conventions
1505 ===========================
1507 vhost-user backends can provide various devices & services and may
1508 need to be configured manually depending on the use case. However, it
1509 is a good idea to follow the conventions listed here when
1510 possible. Users, QEMU or libvirt, can then rely on some common
1511 behaviour to avoid heterogeneous configuration and management of the
1512 backend programs and facilitate interoperability.
1514 Each backend installed on a host system should come with at least one
1515 JSON file that conforms to the vhost-user.json schema. Each file
1516 informs the management applications about the backend type, and binary
1517 location. In addition, it defines rules for management apps for
1518 picking the highest priority backend when multiple match the search
1519 criteria (see ``@VhostUserBackend`` documentation in the schema file).
1521 If the backend is not capable of enabling a requested feature on the
1522 host (such as 3D acceleration with virgl), or the initialization
1523 failed, the backend should fail to start early and exit with a status
1524 != 0. It may also print a message to stderr for further details.
1526 The backend program must not daemonize itself, but it may be
1527 daemonized by the management layer. It may also have a restricted
1528 access to the system.
1530 File descriptors 0, 1 and 2 will exist, and have regular
1531 stdin/stdout/stderr usage (they may have been redirected to /dev/null
1532 by the management layer, or to a log handler).
1534 The backend program must end (as quickly and cleanly as possible) when
1535 the SIGTERM signal is received. Eventually, it may receive SIGKILL by
1536 the management layer after a few seconds.
1538 The following command line options have an expected behaviour. They
1539 are mandatory, unless explicitly said differently:
1543 This option specify the location of the vhost-user Unix domain socket.
1544 It is incompatible with --fd.
1548 When this argument is given, the backend program is started with the
1549 vhost-user socket as file descriptor FDNUM. It is incompatible with
1552 --print-capabilities
1554 Output to stdout the backend capabilities in JSON format, and then
1555 exit successfully. Other options and arguments should be ignored, and
1556 the backend program should not perform its normal function. The
1557 capabilities can be reported dynamically depending on the host
1560 The JSON output is described in the ``vhost-user.json`` schema, by
1561 ```@VHostUserBackendCapabilities``. Example:
1576 Command line options:
1580 Specify the linux input device.
1586 Do no request exclusive access to the input device.
1593 Command line options:
1597 Specify the GPU DRM render node.
1603 Enable virgl rendering support.
1610 Command line options:
1614 Specify block device or file path.