6 This document describes the testing infrastructure in QEMU.
8 Testing with "make check"
9 -------------------------
11 The "make check" testing family includes most of the C based tests in QEMU. For
12 a quick help, run ``make check-help`` from the source tree.
14 The usual way to run these tests is:
20 which includes QAPI schema tests, unit tests, QTests and some iotests.
21 Different sub-types of "make check" tests will be explained below.
23 Before running tests, it is best to build QEMU programs first. Some tests
24 expect the executables to exist and will fail with obscure messages if they
30 Unit tests, which can be invoked with ``make check-unit``, are simple C tests
31 that typically link to individual QEMU object files and exercise them by
32 calling exported functions.
34 If you are writing new code in QEMU, consider adding a unit test, especially
35 for utility modules that are relatively stateless or have few dependencies. To
38 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
40 2. Write the test. Normally you would include the header file which exports
41 the module API, then verify the interface behaves as expected from your
42 test. The test code should be organized with the glib testing framework.
43 Copying and modifying an existing test is usually a good idea.
45 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
46 dictionary called ``tests``. The values are any additional sources and
47 dependencies to be linked with the test. For a simple test whose source
48 is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
56 Since unit tests don't require environment variables, the simplest way to debug
57 a unit test failure is often directly invoking it or even running it under
58 ``gdb``. However there can still be differences in behavior between ``make``
59 invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
60 variable (which affects memory reclamation and catches invalid pointers better)
61 and gtester options. If necessary, you can run
67 and copy the actual command line which executes the unit test, then run
68 it from the command line.
73 QTest is a device emulation testing framework. It can be very useful to test
74 device models; it could also control certain aspects of QEMU (such as virtual
75 clock stepping), with a special purpose "qtest" protocol. Refer to
76 :doc:`qtest` for more details.
78 QTest cases can be executed with
84 Writing portable test cases
85 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
86 Both unit tests and qtests can run on POSIX hosts as well as Windows hosts.
87 Care must be taken when writing portable test cases that can be built and run
88 successfully on various hosts. The following list shows some best practices:
90 * Use portable APIs from glib whenever necessary, e.g.: g_setenv(),
91 g_mkdtemp(), g_mkdir().
92 * Avoid using hardcoded /tmp for temporary file directory.
93 Use g_get_tmp_dir() instead.
94 * Bear in mind that Windows has different special string representation for
95 stdin/stdout/stderr and null devices. For example if your test case uses
96 "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on
97 Windows instead. Also IO redirection does not work on Windows, so avoid
98 using "2>nul" whenever necessary.
99 * If your test cases uses the blkdebug feature, use relative path to pass
100 the config and image file paths in the command line as Windows absolute
101 path contains the delimiter ":" which will confuse the blkdebug parser.
102 * Use double quotes in your extra QEMU command line in your test cases
103 instead of single quotes, as Windows does not drop single quotes when
104 passing the command line to QEMU.
105 * Windows opens a file in text mode by default, while a POSIX compliant
106 implementation treats text files and binary files the same. So if your
107 test cases opens a file to write some data and later wants to compare the
108 written data with the original one, be sure to pass the letter 'b' as
109 part of the mode string to fopen(), or O_BINARY flag for the open() call.
110 * If a certain test case can only run on POSIX or Linux hosts, use a proper
111 #ifdef in the codes. If the whole test suite cannot run on Windows, disable
112 the build in the meson.build file.
117 The QAPI schema tests validate the QAPI parser used by QMP, by feeding
118 predefined input to the parser and comparing the result with the reference
121 The input/output data is managed under the ``tests/qapi-schema`` directory.
122 Each test case includes four files that have a common base name:
124 * ``${casename}.json`` - the file contains the JSON input for feeding the
126 * ``${casename}.out`` - the file contains the expected stdout from the parser
127 * ``${casename}.err`` - the file contains the expected stderr from the parser
128 * ``${casename}.exit`` - the expected error code
130 Consider adding a new QAPI schema test when you are making a change on the QAPI
131 parser (either fixing a bug or extending/modifying the syntax). To do this:
133 1. Add four files for the new case as explained above. For example:
135 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
137 2. Add the new test in ``tests/Makefile.include``. For example:
139 ``qapi-schema += foo.json``
144 ``make check-block`` runs a subset of the block layer iotests (the tests that
145 are in the "auto" group).
146 See the "QEMU iotests" section below for more information.
151 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
152 framework widely used to test block layer related features. It is higher level
153 than "make check" tests and 99% of the code is written in bash or Python
154 scripts. The testing success criteria is golden output comparison, and the
155 test files are named with numbers.
157 To run iotests, make sure QEMU is built successfully, then switch to the
158 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
159 with desired arguments from there.
161 By default, "raw" format and "file" protocol is used; all tests will be
162 executed, except the unsupported ones. You can override the format and protocol
167 # test with qcow2 format
169 # or test a different protocol
172 It's also possible to list test numbers explicitly:
176 # run selected cases with qcow2 format
177 ./check -qcow2 001 030 153
179 Cache mode can be selected with the "-c" option, which may help reveal bugs
180 that are specific to certain cache mode.
182 More options are supported by the ``./check`` script, run ``./check -h`` for
185 Writing a new test case
186 ~~~~~~~~~~~~~~~~~~~~~~~
188 Consider writing a tests case when you are making any changes to the block
189 layer. An iotest case is usually the choice for that. There are already many
190 test cases, so it is possible that extending one of them may achieve the goal
191 and save the boilerplate to create one. (Unfortunately, there isn't a 100%
192 reliable way to find a related one out of hundreds of tests. One approach is
195 Usually an iotest case consists of two files. One is an executable that
196 produces output to stdout and stderr, the other is the expected reference
197 output. They are given the same number in file names. E.g. Test script ``055``
198 and reference output ``055.out``.
200 In rare cases, when outputs differ between cache mode ``none`` and others, a
201 ``.out.nocache`` file is added. In other cases, when outputs differ between
202 image formats, more than one ``.out`` files are created ending with the
203 respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
205 There isn't a hard rule about how to write a test script, but a new test is
206 usually a (copy and) modification of an existing case. There are a few
207 commonly used ways to create a test:
209 * A Bash script. It will make use of several environmental variables related
210 to the testing procedure, and could source a group of ``common.*`` libraries
211 for some common helper routines.
213 * A Python unittest script. Import ``iotests`` and create a subclass of
214 ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
215 this approach is that the output is too scarce, and the script is considered
218 * A simple Python script without using unittest module. This could also import
219 ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
220 from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
221 execution. This is a combination of 1 and 2.
223 Pick the language per your preference since both Bash and Python have
224 comparable library support for invoking and interacting with QEMU programs. If
225 you opt for Python, it is strongly recommended to write Python 3 compatible
228 Both Python and Bash frameworks in iotests provide helpers to manage test
229 images. They can be used to create and clean up images under the test
230 directory. If no I/O or any protocol specific feature is needed, it is often
231 more convenient to use the pseudo block driver, ``null-co://``, as the test
232 image, which doesn't require image creation or cleaning up. Avoid system-wide
233 devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``.
234 Otherwise, image locking implications have to be considered. For example,
235 another application on the host may have locked the file, possibly leading to a
236 test failure. If using such devices are explicitly desired, consider adding
237 ``locking=off`` option to disable image locking.
239 Debugging a test case
240 ~~~~~~~~~~~~~~~~~~~~~
242 The following options to the ``check`` script can be useful when debugging
245 * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
246 connection from a gdb client. The options given to ``gdbserver`` (e.g. the
247 address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
248 environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on
250 It is possible to connect to it for example with
251 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
252 ``gdbserver`` listens on.
253 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
254 regardless of whether it is set or not.
256 * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
257 warnings, it will print and save the log in
258 ``$TEST_DIR/<valgrind_pid>.valgrind``.
259 The final command line will be ``valgrind --log-file=$TEST_DIR/
260 <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
262 * ``-d`` (debug) just increases the logging verbosity, showing
263 for example the QMP commands and answers.
265 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
266 instead of saving it into a log file in
267 ``$TEST_DIR/qemu-machine-<random_string>``.
272 "Tests may belong to one or more test groups, which are defined in the form
273 of a comment in the test source file. By convention, test groups are listed
274 in the second line of the test file, after the "#!/..." line, like this:
278 #!/usr/bin/env python3
283 Another way of defining groups is creating the tests/qemu-iotests/group.local
284 file. This should be used only for downstream (this file should never appear
285 in upstream). This file may be used for defining some downstream test groups
286 or for temporarily disabling tests, like this:
290 # groups for some company downstream process
292 # ci - tests to run on build
293 # down - our downstream tests, not for upstream
295 # Format of each line is:
296 # TEST_NAME TEST_GROUP [TEST_GROUP ]...
301 our-ugly-workaround-test down ci
303 Note that the following group names have a special meaning:
305 - quick: Tests in this group should finish within a few seconds.
307 - auto: Tests in this group are used during "make check" and should be
308 runnable in any case. That means they should run with every QEMU binary
309 (also non-x86), with every QEMU configuration (i.e. must not fail if
310 an optional feature is not compiled in - but reporting a "skip" is ok),
311 work at least with the qcow2 file format, work with all kind of host
312 filesystems and users (e.g. "nobody" or "root") and must not take too
313 much memory and disk space (since CI pipelines tend to fail otherwise).
315 - disabled: Tests in this group are disabled and ignored by check.
319 Container based tests
320 ---------------------
325 The container testing framework in QEMU utilizes public images to
326 build and test QEMU in predefined and widely accessible Linux
327 environments. This makes it possible to expand the test coverage
328 across distros, toolchain flavors and library versions. The support
329 was originally written for Docker although we also support Podman as
330 an alternative container runtime. Although many of the target
331 names and scripts are prefixed with "docker" the system will
332 automatically run on whichever is configured.
334 The container images are also used to augment the generation of tests
335 for testing TCG. See :ref:`checktcg-ref` for more details.
340 Install "docker" with the system package manager and start the Docker service
341 on your development machine, then make sure you have the privilege to run
342 Docker commands. Typically it means setting up passwordless ``sudo docker``
343 command or login as root. For example:
347 $ sudo yum install docker
348 $ # or `apt-get install docker` for Ubuntu, etc.
349 $ sudo systemctl start docker
352 The last command should print an empty table, to verify the system is ready.
354 An alternative method to set up permissions is by adding the current user to
355 "docker" group and making the docker daemon socket file (by default
356 ``/var/run/docker.sock``) accessible to the group:
360 $ sudo groupadd docker
361 $ sudo usermod $USER -a -G docker
362 $ sudo chown :docker /var/run/docker.sock
364 Note that any one of above configurations makes it possible for the user to
365 exploit the whole host with Docker bind mounting or other privileged
366 operations. So only do it on development machines.
371 Install "podman" with the system package manager.
375 $ sudo dnf install podman
378 The last command should print an empty table, to verify the system is ready.
383 From source tree, type ``make docker-help`` to see the help. Testing
384 can be started without configuring or building QEMU (``configure`` and
385 ``make`` are done in the container, with parameters defined by the
390 make docker-test-build@centos8
392 This will create a container instance using the ``centos8`` image (the image
393 is downloaded and initialized automatically), in which the ``test-build`` job
399 The QEMU project has a container registry hosted by GitLab at
400 ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
401 used to pull in pre-built layers. This avoids unnecessary strain on
402 the distro archives created by multiple developers running the same
403 container build steps over and over again. This can be overridden
404 locally by using the ``NOCACHE`` build option:
408 make docker-image-debian-arm64-cross NOCACHE=1
413 Along with many other images, the ``centos8`` image is defined in a Dockerfile
414 in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
415 command will list all the available images.
417 A ``.pre`` script can be added beside the ``.docker`` file, which will be
418 executed before building the image under the build context directory. This is
419 mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
420 for example, to make qemu-user powered cross build containers work.
422 Most of the existing Dockerfiles were written by hand, simply by creating a
423 a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
424 This has led to an inconsistent set of packages being present across the
425 different containers.
427 Thus going forward, QEMU is aiming to automatically generate the Dockerfiles
428 using the ``lcitool`` program provided by the ``libvirt-ci`` project:
430 https://gitlab.com/libvirt/libvirt-ci
432 ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
433 mappings to distribution package names for a wide variety of third
434 party projects. ``lcitool`` applies the mappings to a list of build
435 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
436 list of native packages to install on each distribution, and uses them
437 to generate build environments (dockerfiles and Cirrus CI variable files)
438 that are consistent across OS distribution.
441 Adding new build pre-requisites
442 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
444 When preparing a patch series that adds a new build
445 pre-requisite to QEMU, the prerequisites should to be added to
446 ``tests/lcitool/projects/qemu.yml`` in order to make the dependency
447 available in the CI build environments.
449 In the simple case where the pre-requisite is already known to ``libvirt-ci``
450 the following steps are needed:
452 * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
454 * Run ``make lcitool-refresh`` to re-generate all relevant build environment
457 It may be that ``libvirt-ci`` does not know about the new pre-requisite.
458 If that is the case, some extra preparation steps will be required
459 first to contribute the mapping to the ``libvirt-ci`` project:
461 * Fork the ``libvirt-ci`` project on gitlab
463 * Add an entry for the new build prerequisite to
464 ``lcitool/facts/mappings.yml``, listing its native package name on as
465 many OS distros as practical. Run ``python -m pytest --regenerate-output``
466 and check that the changes are correct.
468 * Commit the ``mappings.yml`` change together with the regenerated test
469 files, and submit a merge request to the ``libvirt-ci`` project.
470 Please note in the description that this is a new build pre-requisite
471 desired for use with QEMU.
473 * CI pipeline will run to validate that the changes to ``mappings.yml``
474 are correct, by attempting to install the newly listed package on
475 all OS distributions supported by ``libvirt-ci``.
477 * Once the merge request is accepted, go back to QEMU and update
478 the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
479 contains the ``mappings.yml`` update. Then add the prerequisite and
480 run ``make lcitool-refresh``.
482 * Please also trigger gitlab container generation pipelines on your change
483 for as many OS distros as practical to make sure that there are no
484 obvious breakages when adding the new pre-requisite. Please see
485 `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation
486 page on how to trigger gitlab CI pipelines on your change.
488 * Please also trigger gitlab container generation pipelines on your change
489 for as many OS distros as practical to make sure that there are no
490 obvious breakages when adding the new pre-requisite. Please see
491 `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation
492 page on how to trigger gitlab CI pipelines on your change.
494 For enterprise distros that default to old, end-of-life versions of the
495 Python runtime, QEMU uses a separate set of mappings that work with more
496 recent versions. These can be found in ``tests/lcitool/mappings.yml``.
497 Modifying this file should not be necessary unless the new pre-requisite
498 is a Python library or tool.
501 Adding new OS distros
502 ^^^^^^^^^^^^^^^^^^^^^
504 In some cases ``libvirt-ci`` will not know about the OS distro that is
505 desired to be tested. Before adding a new OS distro, discuss the proposed
508 * Send a mail to qemu-devel, copying people listed in the
509 MAINTAINERS file for ``Build and test automation``.
511 There are limited CI compute resources available to QEMU, so the
512 cost/benefit tradeoff of adding new OS distros needs to be considered.
514 * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
515 pointing to the qemu-devel mail thread in the archives.
517 This alerts other people who might be interested in the work
518 to avoid duplication, as well as to get feedback from libvirt-ci
519 maintainers on any tips to ease the addition
521 Assuming there is agreement to add a new OS distro then
523 * Fork the ``libvirt-ci`` project on gitlab
525 * Add metadata under ``lcitool/facts/targets/`` for the new OS
526 distro. There might be code changes required if the OS distro
527 uses a package format not currently known. The ``libvirt-ci``
528 maintainers can advise on this when the issue is filed.
530 * Edit the ``lcitool/facts/mappings.yml`` change to add entries for
531 the new OS, listing the native package names for as many packages
532 as practical. Run ``python -m pytest --regenerate-output`` and
533 check that the changes are correct.
535 * Commit the changes to ``lcitool/facts`` and the regenerated test
536 files, and submit a merge request to the ``libvirt-ci`` project.
537 Please note in the description that this is a new build pre-requisite
538 desired for use with QEMU
540 * CI pipeline will run to validate that the changes to ``mappings.yml``
541 are correct, by attempting to install the newly listed package on
542 all OS distributions supported by ``libvirt-ci``.
544 * Once the merge request is accepted, go back to QEMU and update
545 the ``libvirt-ci`` submodule to point to a commit that contains
546 the ``mappings.yml`` update.
552 Different tests are added to cover various configurations to build and test
553 QEMU. Docker tests are the executables under ``tests/docker`` named
554 ``test-*``. They are typically shell scripts and are built on top of a shell
555 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
558 The full list of tests is printed in the ``make docker-help`` help.
560 Debugging a Docker test failure
561 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
563 When CI tasks, maintainers or yourself report a Docker test failure, follow the
564 below steps to debug it:
566 1. Locally reproduce the failure with the reported command line. E.g. run
567 ``make docker-test-mingw@fedora-win64-cross J=8``.
568 2. Add "V=1" to the command line, try again, to see the verbose output.
569 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
570 in the container right before testing starts. You could either manually
571 build QEMU and run tests from there, or press Ctrl-D to let the Docker
573 4. If you press Ctrl-D, the same building and testing procedure will begin, and
574 will hopefully run into the error again. After that, you will be dropped to
575 the prompt for debug.
580 Various options can be used to affect how Docker tests are done. The full
581 list is in the ``make docker`` help text. The frequently used ones are:
583 * ``V=1``: the same as in top level ``make``. It will be propagated to the
584 container and enable verbose output.
585 * ``J=$N``: the number of parallel tasks in make commands in the container,
586 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
587 top level ``make`` will not be propagated into the container.)
588 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
594 Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
595 building and testing with this tool.
597 For more information on TSan:
599 https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
601 Thread Sanitizer in Docker
602 ~~~~~~~~~~~~~~~~~~~~~~~~~~
603 TSan is currently supported in the ubuntu2204 docker.
605 The test-tsan test will build using TSan and then run make check.
609 make docker-test-tsan@ubuntu2204
611 TSan warnings under docker are placed in files located at build/tsan/.
613 We recommend using DEBUG=1 to allow launching the test from inside the docker,
614 and to allow review of the warnings generated by TSan.
616 Building and Testing with TSan
617 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
619 It is possible to build and test with TSan, with a few additional steps.
620 These steps are normally done automatically in the docker.
622 There is a one time patch needed in clang-9 or clang-10 at this time:
626 sed -i 's/^const/static const/g' \
627 /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
629 To configure the build for TSan:
633 ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \
634 --disable-werror --extra-cflags="-O0"
636 The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
639 More information on the TSAN_OPTIONS can be found here:
641 https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
647 export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
648 detect_deadlocks=false history_size=7 exitcode=0 \
649 log_path=<build path>/tsan/tsan_warning
651 The above exitcode=0 has TSan continue without error if any warnings are found.
652 This allows for running the test and then checking the warnings afterwards.
653 If you want TSan to stop and exit with error on warnings, use exitcode=66.
657 Keep in mind that for any data race warning, although there might be a data race
658 detected by TSan, there might be no actual bug here. TSan provides several
659 different mechanisms for suppressing warnings. In general it is recommended
660 to fix the code if possible to eliminate the data race rather than suppress
663 A few important files for suppressing warnings are:
665 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
666 The comment on each suppression will typically indicate why we are
667 suppressing it. More information on the file format can be found here:
669 https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
671 tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable
672 at compile time for test or debug.
673 Add flags to configure to enable:
675 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan"
677 More information on the file format can be found here under "Blacklist Format":
679 https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
683 include/qemu/tsan.h defines annotations. See this file for more descriptions
684 of the annotations themselves. Annotations can be used to suppress
685 TSan warnings or give TSan more information so that it can detect proper
686 relationships between accesses of data.
688 Annotation examples can be found here:
690 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
692 Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp
694 The full set of annotations can be found here:
696 https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
698 docker-binfmt-image-debian-% targets
699 ------------------------------------
701 It is possible to combine Debian's bootstrap scripts with a configured
702 ``binfmt_misc`` to bootstrap a number of Debian's distros including
703 experimental ports not yet supported by a released OS. This can
704 simplify setting up a rootfs by using docker to contain the foreign
705 rootfs rather than manually invoking chroot.
707 Setting up ``binfmt_misc``
708 ~~~~~~~~~~~~~~~~~~~~~~~~~~
710 You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
711 user binary to automatically run binaries for the foreign
712 architecture. While the scripts will try their best to work with
713 dynamically linked QEMU's a statically linked one will present less
714 potential complications when copying into the docker image. Modern
715 kernels support the ``F`` (fix binary) flag which will open the QEMU
716 executable on setup and avoids the need to find and re-open in the
717 chroot environment. This is triggered with the ``--persistent`` flag.
722 For example to setup the HPPA ports builds of Debian::
724 make docker-binfmt-image-debian-sid-hppa \
725 DEB_TYPE=sid DEB_ARCH=hppa \
726 DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
727 DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
728 EXECUTABLE=(pwd)/qemu-hppa V=1
730 The ``DEB_`` variables are substitutions used by
731 ``debian-boostrap.pre`` which is called to do the initial debootstrap
732 of the rootfs before it is copied into the container. The second stage
733 is run as part of the build. The final image will be tagged as
734 ``qemu/debian-sid-hppa``.
739 This test suite contains scripts that bootstrap various guest images that have
740 necessary packages to build QEMU. The basic usage is documented in ``Makefile``
741 help which is displayed with ``make vm-help``.
746 Run ``make vm-help`` to list available make targets. Invoke a specific make
747 command to run build test in an image. For example, ``make vm-build-freebsd``
748 will build the source tree in the FreeBSD image. The command can be executed
749 from either the source tree or the build dir; if the former, ``./configure`` is
750 not needed. The command will then generate the test image in ``./tests/vm/``
751 under the working directory.
753 Note: images created by the scripts accept a well-known RSA key pair for SSH
754 access, so they SHOULD NOT be exposed to external interfaces if you are
755 concerned about attackers taking control of the guest and potentially
756 exploiting a QEMU security bug to compromise the host.
761 By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
762 there isn't one, or if it is older than 2.10, the test won't work. In this case,
763 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
765 Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
770 The ``-j$X`` option in the make command line is not propagated into the VM,
771 specify ``J=$X`` to control the make jobs in the guest.
776 Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
777 debugging and verbose output. If this is not enough, see the next section.
778 ``V=1`` will be propagated down into the make jobs in the guest.
783 Each guest script is an executable script with the same command line options.
784 For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
788 $ cd $QEMU_SRC/tests/vm
790 # To bootstrap the image
791 $ ./netbsd --build-image --image /var/tmp/netbsd.img
794 # To run an arbitrary command in guest (the output will not be echoed unless
796 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
798 # To build QEMU in guest
799 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
801 # To get to an interactive shell
802 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
807 Please look at existing guest scripts for how to add new guests.
809 Most importantly, create a subclass of BaseVM and implement ``build_image()``
810 method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
811 the script's ``main()``.
813 * Usually in ``build_image()``, a template image is downloaded from a
814 predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
815 the checksum, so consider using it.
817 * Once the image is downloaded, users, SSH server and QEMU build deps should
820 - Root password set to ``BaseVM.ROOT_PASS``
821 - User ``BaseVM.GUEST_USER`` is created, and password set to
822 ``BaseVM.GUEST_PASS``
823 - SSH service is enabled and started on boot,
824 ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
825 file of both root and the normal user
826 - DHCP client service is enabled and started on boot, so that it can
827 automatically configure the virtio-net-pci NIC and communicate with QEMU
829 - Necessary packages are installed to untar the source tarball and build
832 * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
833 untars a raw virtio-blk block device, which is the tarball data blob of the
834 QEMU source tree, then configure/build it. Running "make check" is also
840 An image fuzzer was added to exercise format drivers. Currently only qcow2 is
841 supported. To start the fuzzer, run
845 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
847 Alternatively, some command different from ``qemu-img info`` can be tested, by
848 changing the ``-c`` option.
850 Integration tests using the Avocado Framework
851 ---------------------------------------------
853 The ``tests/avocado`` directory hosts integration tests. They're usually
854 higher level tests, and may interact with external resources and with
855 various guest operating systems.
857 These tests are written using the Avocado Testing Framework (which must
858 be installed separately) in conjunction with a the ``avocado_qemu.Test``
859 class, implemented at ``tests/avocado/avocado_qemu``.
861 Tests based on ``avocado_qemu.Test`` can easily:
863 * Customize the command line arguments given to the convenience
864 ``self.vm`` attribute (a QEMUMachine instance)
866 * Interact with the QEMU monitor, send QMP commands and check
869 * Interact with the guest OS, using the convenience console device
870 (which may be useful to assert the effectiveness and correctness of
871 command line arguments or QMP commands)
873 * Interact with external data files that accompany the test itself
874 (see ``self.get_data()``)
876 * Download (and cache) remote data files, such as firmware and kernel
879 * Have access to a library of guest OS images (by means of the
880 ``avocado.utils.vmimage`` library)
882 * Make use of various other test related utilities available at the
883 test class itself and at the utility library:
885 - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
886 - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
891 You can run the avocado tests simply by executing:
897 This involves the automatic installation, from PyPI, of all the
898 necessary avocado-framework dependencies into the QEMU venv within the
899 build tree (at ``./pyvenv``). Test results are also saved within the
900 build tree (at ``tests/results``).
902 Note: the build environment must be using a Python 3 stack, and have
903 the ``venv`` and ``pip`` packages installed. If necessary, make sure
904 ``configure`` is called with ``--python=`` and that those modules are
905 available. On Debian and Ubuntu based systems, depending on the
906 specific version, they may be on packages named ``python3-venv`` and
909 It is also possible to run tests based on tags using the
910 ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
915 make check-avocado AVOCADO_TAGS=quick
917 Note that tags separated with commas have an AND behavior, while tags
918 separated by spaces have an OR behavior. For more information on Avocado
921 https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
923 To run a single test file, a couple of them, or a test within a file
924 using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
925 environment variable with the test files or test names. To run all
926 tests from a single file, use:
930 make check-avocado AVOCADO_TESTS=$FILEPATH
932 The same is valid to run tests from multiple test files:
936 make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
938 To run a single test within a file, use:
942 make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
944 The same is valid to run single tests from multiple test files:
948 make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
950 The scripts installed inside the virtual environment may be used
951 without an "activation". For instance, the Avocado test runner
952 may be invoked by running:
956 pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
958 Note that if ``make check-avocado`` was not executed before, it is
959 possible to create the Python virtual environment with the dependencies
966 It is also possible to run tests from a single file or a single test within
967 a test file. To run tests from a single file within the build tree, use:
971 pyvenv/bin/avocado run tests/avocado/$TESTFILE
973 To run a single test within a test file, use:
977 pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
979 Valid test names are visible in the output from any previous execution
980 of Avocado or ``make check-avocado``, and can also be queried using:
984 pyvenv/bin/avocado list tests/avocado
989 To manually install Avocado and its dependencies, run:
993 pip install --user avocado-framework
995 Alternatively, follow the instructions on this link:
997 https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
1002 The ``tests/avocado/avocado_qemu`` directory provides the
1003 ``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
1004 class. Here's a simple usage example:
1008 from avocado_qemu import QemuSystemTest
1011 class Version(QemuSystemTest):
1013 :avocado: tags=quick
1015 def test_qmp_human_info_version(self):
1017 res = self.vm.cmd('human-monitor-command',
1018 command_line='info version')
1019 self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
1021 To execute your test, run:
1025 avocado run version.py
1027 Tests may be classified according to a convention by using docstring
1028 directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests
1029 in the current directory, tagged as "quick", run:
1033 avocado run -t quick .
1035 The ``avocado_qemu.Test`` base test class
1036 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1038 The ``avocado_qemu.Test`` class has a number of characteristics that
1039 are worth being mentioned right away.
1041 First of all, it attempts to give each test a ready to use QEMUMachine
1042 instance, available at ``self.vm``. Because many tests will tweak the
1043 QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
1044 is left to the test writer.
1046 The base test class has also support for tests with more than one
1047 QEMUMachine. The way to get machines is through the ``self.get_vm()``
1048 method which will return a QEMUMachine instance. The ``self.get_vm()``
1049 method accepts arguments that will be passed to the QEMUMachine creation
1050 and also an optional ``name`` attribute so you can identify a specific
1051 machine and get it more than once through the tests methods. A simple
1052 and hypothetical example follows:
1056 from avocado_qemu import QemuSystemTest
1059 class MultipleMachines(QemuSystemTest):
1060 def test_multiple_machines(self):
1061 first_machine = self.get_vm()
1062 second_machine = self.get_vm()
1063 self.get_vm(name='third_machine').launch()
1065 first_machine.launch()
1066 second_machine.launch()
1068 first_res = first_machine.cmd(
1069 'human-monitor-command',
1070 command_line='info version')
1072 second_res = second_machine.cmd(
1073 'human-monitor-command',
1074 command_line='info version')
1076 third_res = self.get_vm(name='third_machine').cmd(
1077 'human-monitor-command',
1078 command_line='info version')
1080 self.assertEquals(first_res, second_res, third_res)
1082 At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
1085 The ``avocado_qemu.LinuxTest`` base test class
1086 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1088 The ``avocado_qemu.LinuxTest`` is further specialization of the
1089 ``avocado_qemu.Test`` class, so it contains all the characteristics of
1090 the later plus some extra features.
1092 First of all, this base class is intended for tests that need to
1093 interact with a fully booted and operational Linux guest. At this
1094 time, it uses a Fedora 31 guest image. The most basic example looks
1099 from avocado_qemu import LinuxTest
1102 class SomeTest(LinuxTest):
1105 self.launch_and_wait()
1106 self.ssh_command('some_command_to_be_run_in_the_guest')
1108 Please refer to tests that use ``avocado_qemu.LinuxTest`` under
1109 ``tests/avocado`` for more examples.
1114 The QEMUMachine API is already widely used in the Python iotests,
1115 device-crash-test and other Python scripts. It's a wrapper around the
1116 execution of a QEMU binary, giving its users:
1118 * the ability to set command line arguments to be given to the QEMU
1121 * a ready to use QMP connection and interface, which can be used to
1122 send commands and inspect its results, as well as asynchronous
1125 * convenience methods to set commonly used command line arguments in
1126 a more succinct and intuitive way
1128 QEMU binary selection
1129 ^^^^^^^^^^^^^^^^^^^^^
1131 The QEMU binary used for the ``self.vm`` QEMUMachine instance will
1132 primarily depend on the value of the ``qemu_bin`` parameter. If it's
1133 not explicitly set, its default value will be the result of a dynamic
1134 probe in the same source tree. A suitable binary will be one that
1135 targets the architecture matching host machine.
1137 Based on this description, test writers will usually rely on one of
1138 the following approaches:
1140 1) Set ``qemu_bin``, and use the given binary
1142 2) Do not set ``qemu_bin``, and use a QEMU binary named like
1143 "qemu-system-${arch}", either in the current
1144 working directory, or in the current source tree.
1146 The resulting ``qemu_bin`` value will be preserved in the
1147 ``avocado_qemu.Test`` as an attribute with the same name.
1155 Besides the attributes and methods that are part of the base
1156 ``avocado.Test`` class, the following attributes are available on any
1157 ``avocado_qemu.Test`` instance.
1162 A QEMUMachine instance, initially configured according to the given
1163 ``qemu_bin`` parameter.
1168 The architecture can be used on different levels of the stack, e.g. by
1169 the framework or by the test itself. At the framework level, it will
1170 currently influence the selection of a QEMU binary (when one is not
1173 Tests are also free to use this attribute value, for their own needs.
1174 A test may, for instance, use the same value when selecting the
1175 architecture of a kernel or disk image to boot a VM with.
1177 The ``arch`` attribute will be set to the test parameter of the same
1178 name. If one is not given explicitly, it will either be set to
1179 ``None``, or, if the test is tagged with one (and only one)
1180 ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
1185 The cpu model that will be set to all QEMUMachine instances created
1188 The ``cpu`` attribute will be set to the test parameter of the same
1189 name. If one is not given explicitly, it will either be set to
1190 ``None ``, or, if the test is tagged with one (and only one)
1191 ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
1196 The machine type that will be set to all QEMUMachine instances created
1199 The ``machine`` attribute will be set to the test parameter of the same
1200 name. If one is not given explicitly, it will either be set to
1201 ``None``, or, if the test is tagged with one (and only one)
1202 ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
1207 The preserved value of the ``qemu_bin`` parameter or the result of the
1208 dynamic probe for a QEMU binary in the current working directory or
1214 Besides the attributes present on the ``avocado_qemu.Test`` base
1215 class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
1220 The name of the Linux distribution used as the guest image for the
1221 test. The name should match the **Provider** column on the list
1222 of images supported by the avocado.utils.vmimage library:
1224 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1229 The version of the Linux distribution as the guest image for the
1230 test. The name should match the **Version** column on the list
1231 of images supported by the avocado.utils.vmimage library:
1233 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1238 The sha256 hash of the guest image file used for the test.
1240 If this value is not set in the code or by a test parameter (with the
1241 same name), no validation on the integrity of the image will be
1247 To understand how Avocado parameters are accessed by tests, and how
1248 they can be passed to tests, please refer to::
1250 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
1252 Parameter values can be easily seen in the log files, and will look
1257 PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
1265 The architecture that will influence the selection of a QEMU binary
1266 (when one is not explicitly given).
1268 Tests are also free to use this parameter value, for their own needs.
1269 A test may, for instance, use the same value when selecting the
1270 architecture of a kernel or disk image to boot a VM with.
1272 This parameter has a direct relation with the ``arch`` attribute. If
1273 not given, it will default to None.
1278 The cpu model that will be set to all QEMUMachine instances created
1284 The machine type that will be set to all QEMUMachine instances created
1290 The exact QEMU binary to be used on QEMUMachine.
1295 Besides the parameters present on the ``avocado_qemu.Test`` base
1296 class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
1301 The name of the Linux distribution used as the guest image for the
1302 test. The name should match the **Provider** column on the list
1303 of images supported by the avocado.utils.vmimage library:
1305 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1310 The version of the Linux distribution as the guest image for the
1311 test. The name should match the **Version** column on the list
1312 of images supported by the avocado.utils.vmimage library:
1314 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
1319 The sha256 hash of the guest image file used for the test.
1321 If this value is not set in the code or by this parameter no
1322 validation on the integrity of the image will be performed.
1327 The Avocado framework provides Python decorators which allow for easily skip
1328 tests running under certain conditions. For example, on the lack of a binary
1329 on the test system or when the running environment is a CI system. For further
1330 information about those decorators, please refer to::
1332 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
1334 While the conditions for skipping tests are often specifics of each one, there
1335 are recurring scenarios identified by the QEMU developers and the use of
1336 environment variables became a kind of standard way to enable/disable tests.
1338 Here is a list of the most used variables:
1340 AVOCADO_ALLOW_LARGE_STORAGE
1341 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
1342 Tests which are going to fetch or produce assets considered *large* are not
1343 going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
1346 The definition of *large* is a bit arbitrary here, but it usually means an
1347 asset which occupies at least 1GB of size on disk when uncompressed.
1349 AVOCADO_ALLOW_UNTRUSTED_CODE
1350 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1351 There are tests which will boot a kernel image or firmware that can be
1352 considered not safe to run on the developer's workstation, thus they are
1353 skipped by default. The definition of *not safe* is also arbitrary but
1354 usually it means a blob which either its source or build process aren't
1357 You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
1358 order to allow tests which make use of those kind of assets.
1360 AVOCADO_TIMEOUT_EXPECTED
1361 ^^^^^^^^^^^^^^^^^^^^^^^^
1362 The Avocado framework has a timeout mechanism which interrupts tests to avoid the
1363 test suite of getting stuck. The timeout value can be set via test parameter or
1364 property defined in the test class, for further details::
1366 https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
1368 Even though the timeout can be set by the test developer, there are some tests
1369 that may not have a well-defined limit of time to finish under certain
1370 conditions. For example, tests that take longer to execute when QEMU is
1371 compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
1372 has been used to determine whether those tests should run or not.
1376 A number of tests are flagged to not run on the GitLab CI. Usually because
1377 they proved to the flaky or there are constraints on the CI environment which
1378 would make them fail. If you encounter a similar situation then use that
1379 variable as shown on the code snippet below to skip the test:
1383 @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
1387 QEMU_TEST_FLAKY_TESTS
1388 ^^^^^^^^^^^^^^^^^^^^^
1389 Some tests are not working reliably and thus are disabled by default.
1390 Set this environment variable to enable them.
1392 Uninstalling Avocado
1393 ~~~~~~~~~~~~~~~~~~~~
1395 If you've followed the manual installation instructions above, you can
1396 easily uninstall Avocado. Start by listing the packages you have
1401 And remove any package you want with::
1403 pip uninstall <package_name>
1405 If you've used ``make check-avocado``, the Python virtual environment where
1406 Avocado is installed will be cleaned up as part of ``make check-clean``.
1410 Testing with "make check-tcg"
1411 -----------------------------
1413 The check-tcg tests are intended for simple smoke tests of both
1414 linux-user and softmmu TCG functionality. However to build test
1415 programs for guest targets you need to have cross compilers available.
1416 If your distribution supports cross compilers you can do something as
1419 apt install gcc-aarch64-linux-gnu
1421 The configure script will automatically pick up their presence.
1422 Sometimes compilers have slightly odd names so the availability of
1423 them can be prompted by passing in the appropriate configure option
1424 for the architecture in question, for example::
1426 $(configure) --cross-cc-aarch64=aarch64-cc
1428 There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
1429 compiler flags are needed to build for a given target.
1431 If you have the ability to run containers as the user the build system
1432 will automatically use them where no system compiler is available. For
1433 architectures where we also support building QEMU we will generally
1434 use the same container to build tests. However there are a number of
1435 additional containers defined that have a minimal cross-build
1436 environment that is only suitable for building test cases. Sometimes
1437 we may use a bleeding edge distribution for compiler features needed
1438 for test cases that aren't yet in the LTS distros we support for QEMU
1441 See :ref:`container-ref` for more details.
1443 Running subset of tests
1444 ~~~~~~~~~~~~~~~~~~~~~~~
1446 You can build the tests for one architecture::
1448 make build-tcg-tests-$TARGET
1452 make run-tcg-tests-$TARGET
1454 Adding ``V=1`` to the invocation will show the details of how to
1455 invoke QEMU for the test which is useful for debugging tests.
1457 TCG test dependencies
1458 ~~~~~~~~~~~~~~~~~~~~~
1460 The TCG tests are deliberately very light on dependencies and are
1461 either totally bare with minimal gcc lib support (for system-mode tests)
1462 or just glibc (for linux-user tests). This is because getting a cross
1463 compiler to work with additional libraries can be challenging.
1468 There are a number of out-of-tree test suites that are used for more
1469 extensive testing of processor features.
1474 The KVM unit tests are designed to run as a Guest OS under KVM but
1475 there is no reason why they can't exercise the TCG as well. It
1476 provides a minimal OS kernel with hooks for enabling the MMU as well
1477 as reporting test results via a special device::
1479 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
1484 The LTP is focused on exercising the syscall interface of a Linux
1485 kernel. It checks that syscalls behave as documented and strives to
1486 exercise as many corner cases as possible. It is a useful test suite
1487 to run to exercise QEMU's linux-user code::
1489 https://linux-test-project.github.io/
1494 ``gcov`` is a GCC tool to analyze the testing coverage by
1495 instrumenting the tested code. To use it, configure QEMU with
1496 ``--enable-gcov`` option and build. Then run the tests as usual.
1498 If you want to gather coverage information on a single test the ``make
1499 clean-gcda`` target can be used to delete any existing coverage
1500 information before running a single test.
1502 You can generate a HTML coverage report by executing ``make
1503 coverage-html`` which will create
1504 ``meson-logs/coveragereport/index.html``.
1506 Further analysis can be conducted by running the ``gcov`` command
1507 directly on the various .gcda output files. Please read the ``gcov``
1508 documentation for more information.