docs/drvbhyve.html.in

   1 <?xml version="1.0" encoding="UTF-8"?>
   2 <!DOCTYPE html>
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4   <body>
   5     <h1>Bhyve driver</h1>
   6
   7     <ul id="toc"></ul>
   8
   9 <p>
  10 Bhyve is a FreeBSD hypervisor. It first appeared in FreeBSD 10.0. However, it's
  11 recommended to keep tracking FreeBSD 10-STABLE to make sure all new features
  12 of bhyve are supported.
  13
  14 In order to enable bhyve on your FreeBSD host, you'll need to load the <code>vmm</code>
  15 kernel module. Additionally, <code>if_tap</code> and <code>if_bridge</code> modules
  16 should be loaded for networking support. Also, <span class="since">since 3.2.0</span> the
  17 <code>virt-host-validate(1)</code> supports the bhyve host validation and could be
  18 used like this:
  19 </p>
  20
  21 <pre>
  22 $ virt-host-validate bhyve
  23  BHYVE: Checking for vmm module                                              : PASS
  24  BHYVE: Checking for if_tap module                                           : PASS
  25  BHYVE: Checking for if_bridge module                                        : PASS
  26  BHYVE: Checking for nmdm module                                             : PASS
  27 $
  28 </pre>
  29
  30 <p>
  31 Additional information on bhyve could be obtained on <a href="http://bhyve.org/">bhyve.org</a>.
  32 </p>
  33
  34 <h2><a id="uri">Connections to the Bhyve driver</a></h2>
  35 <p>
  36 The libvirt bhyve driver is a single-instance privileged driver. Some sample
  37 connection URIs are:
  38 </p>
  39
  40 <pre>
  41 bhyve:///system                     (local access)
  42 bhyve+unix:///system                (local access)
  43 bhyve+ssh://root@example.com/system (remote access, SSH tunnelled)
  44 </pre>
  45
  46 <h2><a id="exconfig">Example guest domain XML configurations</a></h2>
  47
  48 <h3>Example config</h3>
  49 <p>
  50 The bhyve driver in libvirt is in its early stage and under active development. So it supports
  51 only limited number of features bhyve provides.
  52 </p>
  53
  54 <p>
  55 Note: in older libvirt versions, only a single network device and a single
  56 disk device were supported per-domain. However,
  57 <span class="since">since 1.2.6</span> the libvirt bhyve driver supports
  58 up to 31 PCI devices.
  59 </p>
  60
  61 <p>
  62 Note: the Bhyve driver in libvirt will boot whichever device is first. If you
  63 want to install from CD, put the CD device first. If not, put the root HDD
  64 first.
  65 </p>
  66
  67 <p>
  68 Note: Only the SATA bus is supported. Only <code>cdrom</code>- and
  69 <code>disk</code>-type disks are supported.
  70 </p>
  71
  72 <pre>
  73 &lt;domain type='bhyve'&gt;
  74     &lt;name&gt;bhyve&lt;/name&gt;
  75     &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
  76     &lt;memory&gt;219136&lt;/memory&gt;
  77     &lt;currentMemory&gt;219136&lt;/currentMemory&gt;
  78     &lt;vcpu&gt;1&lt;/vcpu&gt;
  79     &lt;os&gt;
  80        &lt;type&gt;hvm&lt;/type&gt;
  81     &lt;/os&gt;
  82     &lt;features&gt;
  83       &lt;apic/&gt;
  84       &lt;acpi/&gt;
  85     &lt;/features&gt;
  86     &lt;clock offset='utc'/&gt;
  87     &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
  88     &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
  89     &lt;on_crash&gt;destroy&lt;/on_crash&gt;
  90     &lt;devices&gt;
  91       &lt;disk type='file'&gt;
  92         &lt;driver name='file' type='raw'/&gt;
  93         &lt;source file='/path/to/bhyve_freebsd.img'/&gt;
  94         &lt;target dev='hda' bus='sata'/&gt;
  95       &lt;/disk&gt;
  96       &lt;disk type='file' device='cdrom'&gt;
  97         &lt;driver name='file' type='raw'/&gt;
  98         &lt;source file='/path/to/cdrom.iso'/&gt;
  99         &lt;target dev='hdc' bus='sata'/&gt;
 100         &lt;readonly/&gt;
 101       &lt;/disk&gt;
 102       &lt;interface type='bridge'&gt;
 103         &lt;model type='virtio'/&gt;
 104         &lt;source bridge="virbr0"/&gt;
 105       &lt;/interface&gt;
 106     &lt;/devices&gt;
 107 &lt;/domain&gt;
 108 </pre>
 109
 110 <p>(The &lt;disk&gt; sections may be swapped in order to install from
 111 <em>cdrom.iso</em>.)</p>
 112
 113 <h3>Example config (Linux guest)</h3>
 114
 115 <p>
 116 Note the addition of &lt;bootloader&gt;.
 117 </p>
 118
 119 <pre>
 120 &lt;domain type='bhyve'&gt;
 121     &lt;name&gt;linux_guest&lt;/name&gt;
 122     &lt;uuid&gt;df3be7e7-a104-11e3-aeb0-50e5492bd3dc&lt;/uuid&gt;
 123     &lt;memory&gt;131072&lt;/memory&gt;
 124     &lt;currentMemory&gt;131072&lt;/currentMemory&gt;
 125     &lt;vcpu&gt;1&lt;/vcpu&gt;
 126     &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
 127     &lt;os&gt;
 128        &lt;type&gt;hvm&lt;/type&gt;
 129     &lt;/os&gt;
 130     &lt;features&gt;
 131       &lt;apic/&gt;
 132       &lt;acpi/&gt;
 133     &lt;/features&gt;
 134     &lt;clock offset='utc'/&gt;
 135     &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
 136     &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
 137     &lt;on_crash&gt;destroy&lt;/on_crash&gt;
 138     &lt;devices&gt;
 139       &lt;disk type='file' device='disk'&gt;
 140         &lt;driver name='file' type='raw'/&gt;
 141         &lt;source file='/path/to/guest_hdd.img'/&gt;
 142         &lt;target dev='hda' bus='sata'/&gt;
 143       &lt;/disk&gt;
 144       &lt;disk type='file' device='cdrom'&gt;
 145         &lt;driver name='file' type='raw'/&gt;
 146         &lt;source file='/path/to/cdrom.iso'/&gt;
 147         &lt;target dev='hdc' bus='sata'/&gt;
 148         &lt;readonly/&gt;
 149       &lt;/disk&gt;
 150       &lt;interface type='bridge'&gt;
 151         &lt;model type='virtio'/&gt;
 152         &lt;source bridge="virbr0"/&gt;
 153       &lt;/interface&gt;
 154     &lt;/devices&gt;
 155 &lt;/domain&gt;
 156 </pre>
 157
 158 <h3>Example config (Linux UEFI guest, VNC, tablet)</h3>
 159
 160 <p>This is an example to boot into Fedora 25 installation:</p>
 161
 162 <pre>
 163 &lt;domain type='bhyve'&gt;
 164     &lt;name&gt;fedora_uefi_vnc_tablet&lt;/name&gt;
 165     &lt;memory unit='G'&gt;4&lt;/memory&gt;
 166     &lt;vcpu&gt;2&lt;/vcpu&gt;
 167     &lt;os&gt;
 168        &lt;type&gt;hvm&lt;/type&gt;
 169        <b>&lt;loader readonly=&quot;yes&quot; type=&quot;pflash&quot;&gt;/usr/local/share/uefi-firmware/BHYVE_UEFI.fd&lt;/loader&gt;</b>
 170     &lt;/os&gt;
 171     &lt;features&gt;
 172       &lt;apic/&gt;
 173       &lt;acpi/&gt;
 174     &lt;/features&gt;
 175     &lt;clock offset='utc'/&gt;
 176     &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
 177     &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
 178     &lt;on_crash&gt;destroy&lt;/on_crash&gt;
 179     &lt;devices&gt;
 180       &lt;disk type='file' device='cdrom'&gt;
 181         &lt;driver name='file' type='raw'/&gt;
 182           &lt;source file='/path/to/Fedora-Workstation-Live-x86_64-25-1.3.iso'/&gt;
 183         &lt;target dev='hdc' bus='sata'/&gt;
 184         &lt;readonly/&gt;
 185       &lt;/disk&gt;
 186       &lt;disk type='file' device='disk'&gt;
 187         &lt;driver name='file' type='raw'/&gt;
 188         &lt;source file='/path/to/linux_uefi.img'/&gt;
 189         &lt;target dev='hda' bus='sata'/&gt;
 190         &lt;/disk&gt;
 191       &lt;interface type='bridge'&gt;
 192         &lt;model type='virtio'/&gt;
 193         &lt;source bridge=&quot;virbr0&quot;/&gt;
 194       &lt;/interface&gt;
 195       &lt;serial type=&quot;nmdm&quot;&gt;
 196         &lt;source master=&quot;/dev/nmdm0A&quot; slave=&quot;/dev/nmdm0B&quot;/&gt;
 197       &lt;/serial&gt;
 198       <b>&lt;graphics type='vnc' port='5904'&gt;
 199         &lt;listen type='address' address='127.0.0.1'/&gt;
 200       &lt;/graphics&gt;
 201       &lt;controller type='usb' model='nec-xhci'/&gt;
 202       &lt;input type='tablet' bus='usb'/&gt;</b>
 203     &lt;/devices&gt;
 204 &lt;/domain&gt;
 205 </pre>
 206
 207 <p>Please refer to the <a href="#uefi">UEFI</a> section for a more detailed explanation.</p>
 208
 209 <h2><a id="usage">Guest usage / management</a></h2>
 210
 211 <h3><a id="console">Connecting to a guest console</a></h3>
 212
 213 <p>
 214 Guest console connection is supported through the <code>nmdm</code> device. It could be enabled by adding
 215 the following to the domain XML (<span class="since">Since 1.2.4</span>):
 216 </p>
 217
 218 <pre>
 219 ...
 220 &lt;devices&gt;
 221   &lt;serial type="nmdm"&gt;
 222     &lt;source master="/dev/nmdm0A" slave="/dev/nmdm0B"/&gt;
 223   &lt;/serial&gt;
 224 &lt;/devices&gt;
 225 ...</pre>
 226
 227
 228 <p>Make sure to load the <code>nmdm</code> kernel module if you plan to use that.</p>
 229
 230 <p>
 231 Then <code>virsh console</code> command can be used to connect to the text console
 232 of a guest.</p>
 233
 234 <p><b>NB:</b> Some versions of bhyve have a bug that prevents guests from booting
 235 until the console is opened by a client. This bug was fixed in FreeBSD
 236 <a href="http://svnweb.freebsd.org/changeset/base/262884">r262884</a>. If
 237 an older version is used, one either has to open a console manually with <code>virsh console</code>
 238 to let a guest boot or start a guest using:</p>
 239
 240 <pre>start --console domname</pre>
 241
 242 <p><b>NB:</b> A bootloader configured to require user interaction will prevent
 243 the domain from starting (and thus <code>virsh console</code> or <code>start
 244 --console</code> from functioning) until the user interacts with it manually on
 245 the VM host. Because users typically do not have access to the VM host,
 246 interactive bootloaders are unsupported by libvirt. <em>However,</em> if you happen to
 247 run into this scenario and also happen to have access to the Bhyve host
 248 machine, you may select a boot option and allow the domain to finish starting
 249 by using an alternative terminal client on the VM host to connect to the
 250 domain-configured null modem device. One example (assuming
 251 <code>/dev/nmdm0B</code> is configured as the slave end of the domain serial
 252 device) is:</p>
 253
 254 <pre>cu -l /dev/nmdm0B</pre>
 255
 256 <h3><a id="xmltonative">Converting from domain XML to Bhyve args</a></h3>
 257
 258 <p>
 259 The <code>virsh domxml-to-native</code> command can preview the actual
 260 <code>bhyve</code> commands that will be executed for a given domain.
 261 It outputs two lines, the first line is a <code>bhyveload</code> command and
 262 the second is a <code>bhyve</code> command.
 263 </p>
 264
 265 <p>Please note that the <code>virsh domxml-to-native</code> doesn't do any
 266 real actions other than printing the command, for example, it doesn't try to
 267 find a proper TAP interface and create it, like what is done when starting
 268 a domain; and always returns <code>tap0</code> for the network interface. So
 269 if you're going to run these commands manually, most likely you might want to
 270 tweak them.</p>
 271
 272 <pre>
 273 # virsh -c "bhyve:///system"  domxml-to-native --format bhyve-argv --xml /path/to/bhyve.xml
 274 /usr/sbin/bhyveload -m 214 -d /home/user/vm1.img vm1
 275 /usr/sbin/bhyve -c 2 -m 214 -A -I -H -P -s 0:0,hostbridge -s 3:0,virtio-net,tap0,mac=52:54:00:5d:74:e3 -s 2:0,virtio-blk,/home/user/vm1.img -s 1,lpc -l com1,/dev/nmdm0A vm1
 276 </pre>
 277
 278 <h3><a id="zfsvolume">Using ZFS volumes</a></h3>
 279
 280 <p>It's possible to use ZFS volumes as disk devices <span class="since">since 1.2.8</span>.
 281 An example of domain XML device entry for that will look like:</p>
 282
 283 <pre>
 284 ...
 285 &lt;disk type='volume' device='disk'&gt;
 286   &lt;source pool='zfspool' volume='vol1'/&gt;
 287   &lt;target dev='vdb' bus='virtio'/&gt;
 288 &lt;/disk&gt;
 289 ...</pre>
 290
 291 <p>Please refer to the <a href="storage.html">Storage documentation</a> for more details on storage
 292 management.</p>
 293
 294 <h3><a id="grubbhyve">Using grub2-bhyve or Alternative Bootloaders</a></h3>
 295
 296 <p>It's possible to boot non-FreeBSD guests by specifying an explicit
 297 bootloader, e.g. <code>grub-bhyve(1)</code>. Arguments to the bootloader may be
 298 specified as well. If the bootloader is <code>grub-bhyve</code> and arguments
 299 are omitted, libvirt will try and infer boot ordering from user-supplied
 300 &lt;boot order='N'&gt; configuration in the domain. Failing that, it will boot
 301 the first disk in the domain (either <code>cdrom</code>- or
 302 <code>disk</code>-type devices). If the disk type is <code>disk</code>, it will
 303 attempt to boot from the first partition in the disk image.</p>
 304
 305 <pre>
 306 ...
 307 &lt;bootloader&gt;/usr/local/sbin/grub-bhyve&lt;/bootloader&gt;
 308 &lt;bootloader_args&gt;...&lt;/bootloader_args&gt;
 309 ...
 310 </pre>
 311
 312 <p>Caveat: <code>bootloader_args</code> does not support any quoting.
 313 Filenames, etc, must not have spaces or they will be tokenized incorrectly.</p>
 314
 315 <h3><a id="uefi">Using UEFI bootrom, VNC, and USB tablet</a></h3>
 316
 317 <p><span class="since">Since 3.2.0</span>, in addition to <a href="#grubbhyve">grub-bhyve</a>,
 318 non-FreeBSD guests could be also booted using an UEFI boot ROM, provided both guest OS and
 319 installed <code>bhyve(1)</code> version support UEFI. To use that, <code>loader</code>
 320 should be specified in the <code>os</code> section:</p>
 321
 322 <pre>
 323 &lt;domain type='bhyve'&gt;
 324     ...
 325     &lt;os&gt;
 326        &lt;type&gt;hvm&lt;/type&gt;
 327        &lt;loader readonly="yes" type="pflash"&gt;/usr/local/share/uefi-firmware/BHYVE_UEFI.fd&lt;/loader&gt;
 328     &lt;/os&gt;
 329     ...
 330 </pre>
 331
 332 <p>This uses the UEFI firmware provided by
 333 the <a href="https://www.freshports.org/sysutils/bhyve-firmware/">sysutils/bhyve-firmware</a>
 334 FreeBSD port.</p>
 335
 336 <p>VNC and the tablet input device could be configured this way:</p>
 337
 338 <pre>
 339 &lt;domain type='bhyve'&gt;
 340     &lt;devices&gt;
 341       ...
 342       &lt;graphics type='vnc' port='5904'&gt;
 343         &lt;listen type='address' address='127.0.0.1'/&gt;
 344       &lt;/graphics&gt;
 345       &lt;controller type='usb' model='nec-xhci'/&gt;
 346       &lt;input type='tablet' bus='usb'/&gt;
 347     &lt;/devices&gt;
 348     ...
 349 &lt;/domain&gt;
 350 </pre>
 351
 352 <p>This way, VNC will be accessible on <code>127.0.0.1:5904</code>.</p>
 353
 354 <p>Please note that the tablet device requires to have a USB controller
 355 of the <code>nec-xhci</code> model. Currently, only a single controller of this
 356 type and a single tablet are supported per domain.</p>
 357
 358 <p><span class="since">Since 3.5.0</span>, it's possible to configure how the video device is exposed
 359 to the guest using the <code>vgaconf</code> attribute:</p>
 360
 361 <pre>
 362 &lt;domain type='bhyve'&gt;
 363     &lt;devices&gt;
 364     ...
 365       &lt;graphics type='vnc' port='5904'&gt;
 366         &lt;listen type='address' address='127.0.0.1'/&gt;
 367       &lt;/graphics&gt;
 368       &lt;video&gt;
 369         &lt;driver vgaconf='on'/&gt;
 370         &lt;model type='gop' heads='1' primary='yes'/&gt;
 371       &lt;/video&gt;
 372       ...
 373     &lt;/devices&gt;
 374     ...
 375 &lt;/domain&gt;
 376 </pre>
 377
 378 <p>If not specified, bhyve's default mode for <code>vgaconf</code>
 379 will be used. Please refer to the
 380 <a href="https://www.freebsd.org/cgi/man.cgi?query=bhyve&amp;sektion=8&amp;manpath=FreeBSD+12-current">bhyve(8)</a>
 381 manual page and the <a href="https://wiki.freebsd.org/bhyve">bhyve wiki</a> for more details on using
 382 the <code>vgaconf</code> option.</p>
 383
 384 <p><span class="since">Since 3.7.0</span>, it's possible to use <code>autoport</code>
 385 to let libvirt allocate VNC port automatically (instead of explicitly specifying
 386 it with the <code>port</code> attribute):</p>
 387
 388 <pre>
 389     &lt;graphics type='vnc' autoport='yes'&gt;
 390 </pre>
 391
 392 <h3><a id="clockconfig">Clock configuration</a></h3>
 393
 394 <p>Originally bhyve supported only localtime for RTC. Support for UTC time was introduced in
 395 <a href="http://svnweb.freebsd.org/changeset/base/284894">r284894</a> for <i>10-STABLE</i> and
 396 in <a href="http://svnweb.freebsd.org/changeset/base/279225">r279225</a> for <i>-CURRENT</i>.
 397 It's possible to use this in libvirt <span class="since">since 1.2.18</span>, just place the
 398 following to domain XML:</p>
 399
 400 <pre>
 401 &lt;domain type="bhyve"&gt;
 402     ...
 403     &lt;clock offset='utc'/&gt;
 404     ...
 405 &lt;/domain&gt;
 406 </pre>
 407
 408 <p>Please note that if you run the older bhyve version that doesn't support UTC time, you'll
 409 fail to start a domain. As UTC is used as a default when you do not specify clock settings,
 410 you'll need to explicitly specify 'localtime' in this case:</p>
 411
 412 <pre>
 413 &lt;domain type="bhyve"&gt;
 414     ...
 415     &lt;clock offset='localtime'/&gt;
 416     ...
 417 &lt;/domain&gt;
 418 </pre>
 419
 420 <h3><a id="e1000">e1000 NIC</a></h3>
 421
 422 <p>As of <a href="https://svnweb.freebsd.org/changeset/base/302504">r302504</a> bhyve
 423 supports Intel e1000 network adapter emulation. It's supported in libvirt
 424 <span class="since">since 3.1.0</span> and could be used as follows:</p>
 425
 426 <pre>
 427 ...
 428     &lt;interface type='bridge'&gt;
 429       &lt;source bridge='virbr0'/&gt;
 430       &lt;model type='<b>e1000</b>'/&gt;
 431     &lt;/interface&gt;
 432 ...
 433 </pre>
 434
 435 <h3><a id="wired">Wiring guest memory</a></h3>
 436
 437 <p><span class="since">Since 4.4.0</span>, it's possible to specify that guest memory should
 438 be wired and cannot be swapped out as follows:</p>
 439 <pre>
 440 &lt;domain type="bhyve"&gt;
 441     ...
 442     &lt;memoryBacking&gt;
 443       &lt;locked/&gt;
 444     &lt;/memoryBacking&gt;
 445     ...
 446 &lt;/domain&gt;
 447 </pre>
 448
 449 <h3><a id="cputopology">CPU topology</a></h3>
 450
 451 <p><span class="since">Since 4.5.0</span>, it's possible to specify guest CPU topology, if bhyve
 452 supports that. Support for specifying guest CPU topology was added to bhyve in
 453 <a href="http://svnweb.freebsd.org/changeset/base/332298">r332298</a> for <i>-CURRENT</i>.
 454 Example:</p>
 455 <pre>
 456 &lt;domain type="bhyve"&gt;
 457     ...
 458     &lt;cpu&gt;
 459       &lt;topology sockets='1' cores='2' threads='1'/&gt;
 460     &lt;/cpu&gt;
 461     ...
 462 &lt;/domain&gt;
 463 </pre>
 464
 465 <h3><a id="bhyvecommand">Pass-through of arbitrary bhyve commands</a></h3>
 466
 467 <p><span class="since">Since 5.1.0</span>, it's possible to pass additional command-line
 468 arguments to the bhyve process when starting the domain using the
 469 <code>&lt;bhyve:commandline&gt;</code> element under <code>domain</code>.
 470 To supply an argument, use the element <code>&lt;bhyve:arg&gt;</code> with
 471 the attribute <code>value</code> set to additional argument to be added.
 472 The arg element may be repeated multiple times. To use this XML addition, it is necessary
 473 to issue an XML namespace request (the special <code>xmlns:<i>name</i></code> attribute)
 474 that pulls in <code>http://libvirt.org/schemas/domain/bhyve/1.0</code>;
 475 typically, the namespace is given the name of <code>bhyve</code>.
 476 </p>
 477 <p>Example:</p>
 478 <pre>
 479 &lt;domain type="bhyve" xmlns:bhyve="http://libvirt.org/schemas/domain/bhyve/1.0"&gt;
 480   ...
 481   &lt;bhyve:commandline&gt;
 482     &lt;bhyve:arg value='-somebhyvearg'/&gt;
 483   &lt;/bhyve:commandline&gt;
 484 &lt;/domain&gt;
 485 </pre>
 486
 487 <p>Note that these extensions are for testing and development purposes only.
 488 They are <b>unsupported</b>, using them may result in inconsistent state,
 489 and upgrading either bhyve or libvirtd maybe break behavior of a domain that
 490 was relying on a specific commands pass-through.</p>
 491
 492   </body>
 493 </html>