2 Copyright (C) 2017, Emilio G. Cota <cota@braap.org>
3 Copyright (c) 2019, Linaro Limited
4 Written by Emilio Cota and Alex Bennée
11 QEMU TCG plugins provide a way for users to run experiments taking
12 advantage of the total system control emulation can have over a guest.
13 It provides a mechanism for plugins to subscribe to events during
14 translation and execution and optionally callback into the plugin
15 during these events. TCG plugins are unable to change the system state
16 only monitor it passively. However they can do this down to an
17 individual instruction granularity including potentially subscribing
18 to all load and store operations.
23 Any QEMU binary with TCG support has plugins enabled by default.
24 Earlier releases needed to be explicitly enabled with::
26 configure --enable-plugins
28 Once built a program can be run with multiple plugins loaded each with
31 $QEMU $OTHER_QEMU_ARGS \
32 -plugin contrib/plugin/libhowvec.so,inline=on,count=hint \
33 -plugin contrib/plugin/libhotblocks.so
35 Arguments are plugin specific and can be used to modify their
36 behaviour. In this case the howvec plugin is being asked to use inline
37 ops to count and break down the hint instructions by type.
39 Linux user-mode emulation also evaluates the environment variable
42 QEMU_PLUGIN="file=contrib/plugins/libhowvec.so,inline=on,count=hint" $QEMU
50 This is a new feature for QEMU and it does allow people to develop
51 out-of-tree plugins that can be dynamically linked into a running QEMU
52 process. However the project reserves the right to change or break the
53 API should it need to do so. The best way to avoid this is to submit
54 your plugin upstream so they can be updated if/when the API changes.
56 All plugins need to declare a symbol which exports the plugin API
57 version they were built against. This can be done simply by::
59 QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION;
61 The core code will refuse to load a plugin that doesn't export a
62 ``qemu_plugin_version`` symbol or if plugin version is outside of QEMU's
63 supported range of API versions.
65 Additionally the ``qemu_info_t`` structure which is passed to the
66 ``qemu_plugin_install`` method of a plugin will detail the minimum and
67 current API versions supported by QEMU. The API version will be
68 incremented if new APIs are added. The minimum API version will be
69 incremented if existing APIs are changed or removed.
71 Lifetime of the query handle
72 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
74 Each callback provides an opaque anonymous information handle which
75 can usually be further queried to find out information about a
76 translation, instruction or operation. The handles themselves are only
77 valid during the lifetime of the callback so it is important that any
78 information that is needed is extracted during the callback and saved
84 First the plugin is loaded and the public qemu_plugin_install function
85 is called. The plugin will then register callbacks for various plugin
86 events. Generally plugins will register a handler for the *atexit*
87 if they want to dump a summary of collected information once the
88 program/system has finished running.
90 When a registered event occurs the plugin callback is invoked. The
91 callbacks may provide additional information. In the case of a
92 translation event the plugin has an option to enumerate the
93 instructions in a block of instructions and optionally register
94 callbacks to some or all instructions when they are executed.
96 There is also a facility to add an inline event where code to
97 increment a counter can be directly inlined with the translation.
98 Currently only a simple increment is supported. This is not atomic so
99 can miss counts. If you want absolute precision you should use a
100 callback which can then ensure atomicity itself.
102 Finally when QEMU exits all the registered *atexit* callbacks are
105 Exposure of QEMU internals
106 ~~~~~~~~~~~~~~~~~~~~~~~~~~
108 The plugin architecture actively avoids leaking implementation details
109 about how QEMU's translation works to the plugins. While there are
110 conceptions such as translation time and translation blocks the
111 details are opaque to plugins. The plugin is able to query select
112 details of instructions and system configuration only through the
113 exported *qemu_plugin* functions.
115 However the following assumptions can be made:
120 All code will go through a translation phase although not all
121 translations will be necessarily be executed. You need to instrument
122 actual executions to track what is happening.
124 It is quite normal to see the same address translated multiple times.
125 If you want to track the code in system emulation you should examine
126 the underlying physical address (``qemu_plugin_insn_haddr``) to take
127 into account the effects of virtual memory although if the system does
128 paging this will change too.
130 Not all instructions in a block will always execute so if its
131 important to track individual instruction execution you need to
132 instrument them directly. However asynchronous interrupts will not
133 change control flow mid-block.
138 Instruction instrumentation runs before the instruction executes. You
139 can be can be sure the instruction will be dispatched, but you can't
140 be sure it will complete. Generally this will be because of a
141 synchronous exception (e.g. SIGILL) triggered by the instruction
142 attempting to execute. If you want to be sure you will need to
143 instrument the next instruction as well. See the ``execlog.c`` plugin
144 for examples of how to track this and finalise details after execution.
149 Memory callbacks are called after a successful load or store.
150 Unsuccessful operations (i.e. faults) will not be visible to memory
151 instrumentation although the execution side effects can be observed
152 (e.g. entering a exception handler).
154 System Idle and Resume States
155 +++++++++++++++++++++++++++++
157 The ``qemu_plugin_register_vcpu_idle_cb`` and
158 ``qemu_plugin_register_vcpu_resume_cb`` functions can be used to track
159 when CPUs go into and return from sleep states when waiting for
160 external I/O. Be aware though that these may occur less frequently
161 than in real HW due to the inefficiencies of emulation giving less
162 chance for the CPU to idle.
170 We have to ensure we cannot deadlock, particularly under MTTCG. For
171 this we acquire a lock when called from plugin code. We also keep the
172 list of callbacks under RCU so that we do not have to hold the lock
173 when calling the callbacks. This is also for performance, since some
174 callbacks (e.g. memory access callbacks) might be called very
177 * A consequence of this is that we keep our own list of CPUs, so that
178 we do not have to worry about locking order wrt cpu_list_lock.
179 * Use a recursive lock, since we can get registration calls from
182 As a result registering/unregistering callbacks is "slow", since it
183 takes a lock. But this is very infrequent; we want performance when
184 calling (or not calling) callbacks, not when registering them. Using
185 RCU is great for this.
187 We support the uninstallation of a plugin at any time (e.g. from
188 plugin callbacks). This allows plugins to remove themselves if they no
189 longer want to instrument the code. This operation is asynchronous
190 which means callbacks may still occur after the uninstall operation is
191 requested. The plugin isn't completely uninstalled until the safe work
192 has executed while all vCPUs are quiescent.
197 There are a number of plugins included with QEMU and you are
198 encouraged to contribute your own plugins plugins upstream. There is a
199 ``contrib/plugins`` directory where they can go. There are also some
200 basic plugins that are used to test and exercise the API during the
201 ``make check-tcg`` target in ``tests\plugins``.
203 - tests/plugins/empty.c
205 Purely a test plugin for measuring the overhead of the plugins system
206 itself. Does no instrumentation.
210 A very basic plugin which will measure execution in course terms as
211 each basic block is executed. By default the results are shown once
214 $ qemu-aarch64 -plugin tests/plugin/libbb.so \
215 -d plugin ./tests/tcg/aarch64-linux-user/sha1
216 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6
217 bb's: 2277338, insns: 158483046
219 Behaviour can be tweaked with the following arguments:
223 Use faster inline addition of a single counter. Not per-cpu and not
228 Dump the current execution stats whenever the guest vCPU idles
230 - tests/plugins/insn.c
232 This is a basic instruction level instrumentation which can count the
233 number of instructions executed on each core/thread::
235 $ qemu-aarch64 -plugin tests/plugin/libinsn.so \
236 -d plugin ./tests/tcg/aarch64-linux-user/threadcount
249 Behaviour can be tweaked with the following arguments:
253 Use faster inline addition of a single counter. Not per-cpu and not
258 Give a summary of the instruction sizes for the execution
262 Only instrument instructions matching the string prefix. Will show
263 some basic stats including how many instructions have executed since
264 the last execution. For example::
266 $ qemu-aarch64 -plugin tests/plugin/libinsn.so,match=bl \
267 -d plugin ./tests/tcg/aarch64-linux-user/sha512-vector
269 0x40069c, 'bl #0x4002b0', 10 hits, 1093 match hits, Δ+1257 since last match, 98 avg insns/match
270 0x4006ac, 'bl #0x403690', 10 hits, 1094 match hits, Δ+47 since last match, 98 avg insns/match
271 0x4037fc, 'bl #0x4002b0', 18 hits, 1095 match hits, Δ+22 since last match, 98 avg insns/match
272 0x400720, 'bl #0x403690', 10 hits, 1096 match hits, Δ+58 since last match, 98 avg insns/match
273 0x4037fc, 'bl #0x4002b0', 19 hits, 1097 match hits, Δ+22 since last match, 98 avg insns/match
274 0x400730, 'bl #0x403690', 10 hits, 1098 match hits, Δ+33 since last match, 98 avg insns/match
275 0x4037ac, 'bl #0x4002b0', 12 hits, 1099 match hits, Δ+20 since last match, 98 avg insns/match
278 For more detailed execution tracing see the ``execlog`` plugin for
281 - tests/plugins/mem.c
283 Basic instruction level memory instrumentation::
285 $ qemu-aarch64 -plugin tests/plugin/libmem.so,inline=true \
286 -d plugin ./tests/tcg/aarch64-linux-user/sha1
287 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6
288 inline mem accesses: 79525013
290 Behaviour can be tweaked with the following arguments:
294 Use faster inline addition of a single counter. Not per-cpu and not
297 * callback=true|false
299 Use callbacks on each memory instrumentation.
303 Count IO accesses (only for system emulation)
305 - tests/plugins/syscall.c
307 A basic syscall tracing plugin. This only works for user-mode. By
308 default it will give a summary of syscall stats at the end of the
311 $ qemu-aarch64 -plugin tests/plugin/libsyscall \
312 -d plugin ./tests/tcg/aarch64-linux-user/threadcount
315 syscall no. calls errors
335 - contrib/plugins/hotblocks.c
337 The hotblocks plugin allows you to examine the where hot paths of
338 execution are in your program. Once the program has finished you will
339 get a sorted list of blocks reporting the starting PC, translation
340 count, number of instructions and execution count. This will work best
341 with linux-user execution as system emulation tends to generate
342 re-translations as blocks from different programs get swapped in and
343 out of system memory.
345 If your program is single-threaded you can use the ``inline`` option for
346 slightly faster (but not thread safe) counters.
351 -plugin contrib/plugins/libhotblocks.so -d plugin \
352 ./tests/tcg/aarch64-linux-user/sha1
353 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6
354 collected 903 entries in the hash table
355 pc, tcount, icount, ecount
356 0x0000000041ed10, 1, 5, 66087
357 0x000000004002b0, 1, 4, 66087
360 - contrib/plugins/hotpages.c
362 Similar to hotblocks but this time tracks memory accesses::
365 -plugin contrib/plugins/libhotpages.so -d plugin \
366 ./tests/tcg/aarch64-linux-user/sha1
367 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6
368 Addr, RCPUs, Reads, WCPUs, Writes
369 0x000055007fe000, 0x0001, 31747952, 0x0001, 8835161
370 0x000055007ff000, 0x0001, 29001054, 0x0001, 8780625
371 0x00005500800000, 0x0001, 687465, 0x0001, 335857
372 0x0000000048b000, 0x0001, 130594, 0x0001, 355
373 0x0000000048a000, 0x0001, 1826, 0x0001, 11
375 The hotpages plugin can be configured using the following arguments:
377 * sortby=reads|writes|address
379 Log the data sorted by either the number of reads, the number of writes, or
380 memory address. (Default: entries are sorted by the sum of reads and writes)
384 Track IO addresses. Only relevant to full system emulation. (Default: off)
388 The page size used. (Default: N = 4096)
390 - contrib/plugins/howvec.c
392 This is an instruction classifier so can be used to count different
393 types of instructions. It has a number of options to refine which get
394 counted. You can give a value to the ``count`` argument for a class of
395 instructions to break it down fully, so for example to see all the system
398 $ qemu-system-aarch64 $(QEMU_ARGS) \
399 -append "root=/dev/sda2 systemd.unit=benchmark.service" \
400 -smp 4 -plugin ./contrib/plugins/libhowvec.so,count=sreg -d plugin
402 which will lead to a sorted list after the class breakdown::
405 Class: UDEF not counted
407 Class: PCrel addr (47789483 hits)
408 Class: Add/Sub (imm) (192817388 hits)
409 Class: Logical (imm) (93852565 hits)
410 Class: Move Wide (imm) (76398116 hits)
411 Class: Bitfield (44706084 hits)
412 Class: Extract (5499257 hits)
413 Class: Cond Branch (imm) (147202932 hits)
414 Class: Exception Gen (193581 hits)
415 Class: NOP not counted
416 Class: Hints (6652291 hits)
417 Class: Barriers (8001661 hits)
418 Class: PSTATE (1801695 hits)
419 Class: System Insn (6385349 hits)
420 Class: System Reg counted individually
421 Class: Branch (reg) (69497127 hits)
422 Class: Branch (imm) (84393665 hits)
423 Class: Cmp & Branch (110929659 hits)
424 Class: Tst & Branch (44681442 hits)
425 Class: AdvSimd ldstmult (736 hits)
426 Class: ldst excl (9098783 hits)
427 Class: Load Reg (lit) (87189424 hits)
428 Class: ldst noalloc pair (3264433 hits)
429 Class: ldst pair (412526434 hits)
430 Class: ldst reg (imm) (314734576 hits)
431 Class: Loads & Stores (2117774 hits)
432 Class: Data Proc Reg (223519077 hits)
433 Class: Scalar FP (31657954 hits)
434 Individual Instructions:
435 Instr: mrs x0, sp_el0 (2682661 hits) (op=0xd5384100/ System Reg)
436 Instr: mrs x1, tpidr_el2 (1789339 hits) (op=0xd53cd041/ System Reg)
437 Instr: mrs x2, tpidr_el2 (1513494 hits) (op=0xd53cd042/ System Reg)
438 Instr: mrs x0, tpidr_el2 (1490823 hits) (op=0xd53cd040/ System Reg)
439 Instr: mrs x1, sp_el0 (933793 hits) (op=0xd5384101/ System Reg)
440 Instr: mrs x2, sp_el0 (699516 hits) (op=0xd5384102/ System Reg)
441 Instr: mrs x4, tpidr_el2 (528437 hits) (op=0xd53cd044/ System Reg)
442 Instr: mrs x30, ttbr1_el1 (480776 hits) (op=0xd538203e/ System Reg)
443 Instr: msr ttbr1_el1, x30 (480713 hits) (op=0xd518203e/ System Reg)
444 Instr: msr vbar_el1, x30 (480671 hits) (op=0xd518c01e/ System Reg)
447 To find the argument shorthand for the class you need to examine the
448 source code of the plugin at the moment, specifically the ``*opt``
449 argument in the InsnClassExecCount tables.
451 - contrib/plugins/lockstep.c
453 This is a debugging tool for developers who want to find out when and
454 where execution diverges after a subtle change to TCG code generation.
455 It is not an exact science and results are likely to be mixed once
456 asynchronous events are introduced. While the use of -icount can
457 introduce determinism to the execution flow it doesn't always follow
458 the translation sequence will be exactly the same. Typically this is
459 caused by a timer firing to service the GUI causing a block to end
460 early. However in some cases it has proved to be useful in pointing
461 people at roughly where execution diverges. The only argument you need
462 for the plugin is a path for the socket the two instances will
466 $ qemu-system-sparc -monitor none -parallel none \
467 -net none -M SS-20 -m 256 -kernel day11/zImage.elf \
468 -plugin ./contrib/plugins/liblockstep.so,sockpath=lockstep-sparc.sock \
471 which will eventually report::
473 qemu-system-sparc: warning: nic lance.0 has no peer
474 @ 0x000000ffd06678 vs 0x000000ffd001e0 (2/1 since last)
475 @ 0x000000ffd07d9c vs 0x000000ffd06678 (3/1 since last)
476 Δ insn_count @ 0x000000ffd07d9c (809900609) vs 0x000000ffd06678 (809900612)
477 previously @ 0x000000ffd06678/10 (809900609 insns)
478 previously @ 0x000000ffd001e0/4 (809900599 insns)
479 previously @ 0x000000ffd080ac/2 (809900595 insns)
480 previously @ 0x000000ffd08098/5 (809900593 insns)
481 previously @ 0x000000ffd080c0/1 (809900588 insns)
483 - contrib/plugins/hwprofile.c
485 The hwprofile tool can only be used with system emulation and allows
486 the user to see what hardware is accessed how often. It has a number of options:
488 * track=read or track=write
490 By default the plugin tracks both reads and writes. You can use one
491 of these options to limit the tracking to just one class of accesses.
495 Will include a detailed break down of what the guest PC that made the
496 access was. Not compatible with the pattern option. Example output::
498 cirrus-low-memory @ 0xfffffd00000a0000
499 pc:fffffc0000005cdc, 1, 256
500 pc:fffffc0000005ce8, 1, 256
501 pc:fffffc0000005cec, 1, 256
505 Instead break down the accesses based on the offset into the HW
506 region. This can be useful for seeing the most used registers of a
507 device. Example output::
509 pci0-conf @ 0xfffffd01fe000000
518 - contrib/plugins/execlog.c
520 The execlog tool traces executed instructions with memory access. It can be used
521 for debugging and security analysis purposes.
522 Please be aware that this will generate a lot of output.
524 The plugin needs default argument::
526 $ qemu-system-arm $(QEMU_ARGS) \
527 -plugin ./contrib/plugins/libexeclog.so -d plugin
529 which will output an execution trace following this structure::
531 # vCPU, vAddr, opcode, disassembly[, load/store, memory addr, device]...
532 0, 0xa12, 0xf8012400, "movs r4, #0"
533 0, 0xa14, 0xf87f42b4, "cmp r4, r6"
534 0, 0xa16, 0xd206, "bhs #0xa26"
535 0, 0xa18, 0xfff94803, "ldr r0, [pc, #0xc]", load, 0x00010a28, RAM
536 0, 0xa1a, 0xf989f000, "bl #0xd30"
537 0, 0xd30, 0xfff9b510, "push {r4, lr}", store, 0x20003ee0, RAM, store, 0x20003ee4, RAM
538 0, 0xd32, 0xf9893014, "adds r0, #0x14"
539 0, 0xd34, 0xf9c8f000, "bl #0x10c8"
540 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM
542 the output can be filtered to only track certain instructions or
543 addresses using the ``ifilter`` or ``afilter`` options. You can stack the
544 arguments if required::
546 $ qemu-system-arm $(QEMU_ARGS) \
547 -plugin ./contrib/plugins/libexeclog.so,ifilter=st1w,afilter=0x40001808 -d plugin
549 This plugin can also dump registers when they change value. Specify the name of the
550 registers with multiple ``reg`` options. You can also use glob style matching if you wish::
552 $ qemu-system-arm $(QEMU_ARGS) \
553 -plugin ./contrib/plugins/libexeclog.so,reg=\*_el2,reg=sp -d plugin
555 Be aware that each additional register to check will slow down
556 execution quite considerably. You can optimise the number of register
557 checks done by using the rdisas option. This will only instrument
558 instructions that mention the registers in question in disassembly.
559 This is not foolproof as some instructions implicitly change
560 instructions. You can use the ifilter to catch these cases:
562 $ qemu-system-arm $(QEMU_ARGS) \
563 -plugin ./contrib/plugins/libexeclog.so,ifilter=msr,ifilter=blr,reg=x30,reg=\*_el1,rdisas=on
565 - contrib/plugins/cache.c
567 Cache modelling plugin that measures the performance of a given L1 cache
568 configuration, and optionally a unified L2 per-core cache when a given working
571 $ qemu-x86_64 -plugin ./contrib/plugins/libcache.so \
572 -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs
574 will report the following::
576 core #, data accesses, data misses, dmiss rate, insn accesses, insn misses, imiss rate
577 0 996695 508 0.0510% 2642799 18617 0.7044%
579 address, data misses, instruction
580 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx)
581 0x41f395 (_IO_default_xsputn), 49, movb %dl, (%rdi, %rax)
582 0x42584d (ptmalloc_init.part.0), 33, movaps %xmm0, (%rax)
583 0x454d48 (__tunables_init), 20, cmpb $0, (%r8)
586 address, fetch misses, instruction
587 0x4160a0 (__vfprintf_internal), 744, movl $1, %ebx
588 0x41f0a0 (_IO_setb), 744, endbr64
589 0x415882 (__vfprintf_internal), 744, movq %r12, %rdi
590 0x4268a0 (__malloc), 696, andq $0xfffffffffffffff0, %rax
593 The plugin has a number of arguments, all of them are optional:
597 Print top N icache and dcache thrashing instructions along with their
598 address, number of misses, and its disassembly. (default: 32)
604 Instruction cache configuration arguments. They specify the cache size, block
605 size, and associativity of the instruction cache, respectively.
606 (default: N = 16384, B = 64, A = 8)
612 Data cache configuration arguments. They specify the cache size, block size,
613 and associativity of the data cache, respectively.
614 (default: N = 16384, B = 64, A = 8)
618 Sets the eviction policy to POLICY. Available policies are: :code:`lru`,
619 :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for
620 both instruction and data caches. (default: POLICY = :code:`lru`)
624 Sets the number of cores for which we maintain separate icache and dcache.
625 (default: for linux-user, N = 1, for full system emulation: N = cores
630 Simulates a unified L2 cache (stores blocks for both instructions and data)
631 using the default L2 configuration (cache size = 2MB, associativity = 16-way,
638 L2 cache configuration arguments. They specify the cache size, block size, and
639 associativity of the L2 cache, respectively. Setting any of the L2
640 configuration arguments implies ``l2=on``.
641 (default: N = 2097152 (2MB), B = 64, A = 16)
646 The following API is generated from the inline documentation in
647 ``include/qemu/qemu-plugin.h``. Please ensure any updates to the API
648 include the full kernel-doc annotations.
650 .. kernel-doc:: include/qemu/qemu-plugin.h