diff options
Diffstat (limited to '')
| -rw-r--r-- | tools/perf/Documentation/perf-intel-pt.txt | 463 |
1 files changed, 456 insertions, 7 deletions
diff --git a/tools/perf/Documentation/perf-intel-pt.txt b/tools/perf/Documentation/perf-intel-pt.txt index cbb920f5d056..92464a5d7eaf 100644 --- a/tools/perf/Documentation/perf-intel-pt.txt +++ b/tools/perf/Documentation/perf-intel-pt.txt @@ -108,9 +108,10 @@ displayed as follows: perf script --itrace=ibxwpe -F+flags -The flags are "bcrosyiABExgh" which stand for branch, call, return, conditional, +The flags are "bcrosyiABExghDt" which stand for branch, call, return, conditional, system, asynchronous, interrupt, transaction abort, trace begin, trace end, -in transaction, VM-entry, and VM-exit respectively. +in transaction, VM-entry, VM-exit, interrupt disabled, and interrupt disable +toggle respectively. perf script also supports higher level ways to dump instruction traces: @@ -266,7 +267,7 @@ Note that, as with all events, the event is suffixed with event modifiers: H host p precise ip -'h', 'G' and 'H' are for virtualization which is not supported by Intel PT. +'h', 'G' and 'H' are for virtualization which are not used by Intel PT. 'p' is also not relevant to Intel PT. So only options 'u' and 'k' are meaningful for Intel PT. @@ -467,6 +468,8 @@ ptw Enable PTWRITE packets which are produced when a ptwrite instruction which contains "1" if the feature is supported and "0" otherwise. + As an alternative, refer to "Emulated PTWRITE" further below. + fup_on_ptw Enable a FUP packet to follow the PTWRITE packet. The FUP packet provides the address of the ptwrite instruction. In the absence of fup_on_ptw, the decoder will use the address of the previous branch @@ -483,6 +486,30 @@ pwr_evt Enable power events. The power events provide information about which contains "1" if the feature is supported and "0" otherwise. +event Enable Event Trace. The events provide information about asynchronous + events. + + Support for this feature is indicated by: + + /sys/bus/event_source/devices/intel_pt/caps/event_trace + + which contains "1" if the feature is supported and + "0" otherwise. + +notnt Disable TNT packets. Without TNT packets, it is not possible to walk + executable code to reconstruct control flow, however FUP, TIP, TIP.PGE + and TIP.PGD packets still indicate asynchronous control flow, and (if + return compression is disabled - see noretcomp) return statements. + The advantage of eliminating TNT packets is reducing the size of the + trace and corresponding tracing overhead. + + Support for this feature is indicated by: + + /sys/bus/event_source/devices/intel_pt/caps/tnt_disable + + which contains "1" if the feature is supported and + "0" otherwise. + AUX area sampling option ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -876,6 +903,8 @@ The letters are: p synthesize "power" events (incl. PSB events) c synthesize branches events (calls only) r synthesize branches events (returns only) + o synthesize PEBS-via-PT events + I synthesize Event Trace events e synthesize tracing error events d create a debug log g synthesize a call chain (use with i or x) @@ -914,12 +943,15 @@ event packets are recorded only if the "pwr_evt" config term was used. Refer to the config terms section above. The power events record information about C-state changes, whereas CBR is indicative of CPU frequency. perf script "event,synth" fields display information like this: + cbr: cbr: 22 freq: 2189 MHz (200%) mwait: hints: 0x60 extensions: 0x1 pwre: hw: 0 cstate: 2 sub-cstate: 0 exstop: ip: 1 pwrx: deepest cstate: 2 last cstate: 2 wake reason: 0x4 + Where: + "cbr" includes the frequency and the percentage of maximum non-turbo "mwait" shows mwait hints and extensions "pwre" shows C-state transitions (to a C-state deeper than C0) and @@ -927,6 +959,7 @@ Where: "exstop" indicates execution stopped and whether the IP was recorded exactly, "pwrx" indicates return to C0 + For more details refer to the Intel 64 and IA-32 Architectures Software Developer Manuals. @@ -940,8 +973,10 @@ are quite important. Users must know if what they are seeing is a complete picture or not. The "e" option may be followed by flags which affect what errors will or will not be reported. Each flag must be preceded by either '+' or '-'. The flags supported by Intel PT are: + -o Suppress overflow errors -l Suppress trace data lost errors + For example, for errors but not overflow or data lost errors: --itrace=e-o-l @@ -951,11 +986,16 @@ decoded packets and instructions. Note that this option slows down the decoder and that the resulting file may be very large. The "d" option may be followed by flags which affect what debug messages will or will not be logged. Each flag must be preceded by either '+' or '-'. The flags support by Intel PT are: + -a Suppress logging of perf events +a Log all perf events + +e Output only on decoding errors (size configurable) +o Output to stdout instead of "intel_pt.log" + By default, logged perf events are filtered by any specified time ranges, but -flag +a overrides that. +flag +a overrides that. The +e flag can be useful for analyzing errors. By +default, the log size in that case is 16384 bytes, but can be altered by +linkperf:perf-config[1] e.g. perf config itrace.debug-log-buffer-size=30000 In addition, the period of the "instructions" event can be specified. e.g. @@ -1189,10 +1229,10 @@ XED include::build-xed.txt[] -Tracing Virtual Machines ------------------------- +Tracing Virtual Machines (kernel only) +-------------------------------------- -Currently, only kernel tracing is supported and only with either "timeless" decoding +Currently, kernel tracing is supported with either "timeless" decoding (i.e. no TSC timestamps) or VM Time Correlation. VM Time Correlation is an extra step using 'perf inject' and requires unchanging VMX TSC Offset and no VMX TSC Scaling. @@ -1371,6 +1411,415 @@ There were none. :17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax +Tracing Virtual Machines (including user space) +----------------------------------------------- + +It is possible to use perf record to record sideband events within a virtual machine, so that an Intel PT trace on the host can be decoded. +Sideband events from the guest perf.data file can be injected into the host perf.data file using perf inject. + +Here is an example of the steps needed: + +On the guest machine: + +Check that no-kvmclock kernel command line option was used to boot: + +Note, this is essential to enable time correlation between host and guest machines. + + $ cat /proc/cmdline + BOOT_IMAGE=/boot/vmlinuz-5.10.0-16-amd64 root=UUID=cb49c910-e573-47e0-bce7-79e293df8e1d ro no-kvmclock + +There is no BPF support at present so, if possible, disable JIT compiling: + + $ echo 0 | sudo tee /proc/sys/net/core/bpf_jit_enable + 0 + +Start perf record to collect sideband events: + + $ sudo perf record -o guest-sideband-testing-guest-perf.data --sample-identifier --buildid-all --switch-events --kcore -a -e dummy + +On the host machine: + +Start perf record to collect Intel PT trace: + +Note, the host trace will get very big, very fast, so the steps from starting to stopping the host trace really need to be done so that they happen in the shortest time possible. + + $ sudo perf record -o guest-sideband-testing-host-perf.data -m,64M --kcore -a -e intel_pt/cyc/ + +On the guest machine: + +Run a small test case, just 'uname' in this example: + + $ uname + Linux + +On the host machine: + +Stop the Intel PT trace: + + ^C + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 76.122 MB guest-sideband-testing-host-perf.data ] + +On the guest machine: + +Stop the Intel PT trace: + + ^C + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 1.247 MB guest-sideband-testing-guest-perf.data ] + +And then copy guest-sideband-testing-guest-perf.data to the host (not shown here). + +On the host machine: + +With the 2 perf.data recordings, and with their ownership changed to the user. + +Identify the TSC Offset: + + $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=dry-run + VMCS: 0x103fc6 TSC Offset 0xfffffa6ae070cb20 + VMCS: 0x103ff2 TSC Offset 0xfffffa6ae070cb20 + VMCS: 0x10fdaa TSC Offset 0xfffffa6ae070cb20 + VMCS: 0x24d57c TSC Offset 0xfffffa6ae070cb20 + +Correct Intel PT TSC timestamps for the guest machine: + + $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=0xfffffa6ae070cb20 --force + +Identify the guest machine PID: + + $ perf script -i guest-sideband-testing-host-perf.data --no-itrace --show-task-events | grep KVM + CPU 0/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 0/KVM:13376/13381 + CPU 1/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 1/KVM:13376/13382 + CPU 2/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 2/KVM:13376/13383 + CPU 3/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 3/KVM:13376/13384 + +Note, the QEMU option -name debug-threads=on is needed so that thread names +can be used to determine which thread is running which VCPU as above. libvirt seems to use this by default. + +Create a guestmount, assuming the guest machine is 'vm_to_test': + + $ mkdir -p ~/guestmount/13376 + $ sshfs -o direct_io vm_to_test:/ ~/guestmount/13376 + +Inject the guest perf.data file into the host perf.data file: + +Note, due to the guestmount option, guest object files and debug files will be copied into the build ID cache from the guest machine, with the notable exception of VDSO. +If needed, VDSO can be copied manually in a fashion similar to that used by the perf-archive script. + + $ perf inject -i guest-sideband-testing-host-perf.data -o inj --guestmount ~/guestmount --guest-data=guest-sideband-testing-guest-perf.data,13376,0xfffffa6ae070cb20 + +Show an excerpt from the result. In this case the CPU and time range have been to chosen to show interaction between guest and host when 'uname' is starting to run on the guest machine: + +Notes: + + - the CPU displayed, [002] in this case, is always the host CPU + - events happening in the virtual machine start with VM:13376 VCPU:003, which shows the hypervisor PID 13376 and the VCPU number + - only calls and errors are displayed i.e. --itrace=ce + - branches entering and exiting the virtual machine are split, and show as 2 branches to/from "0 [unknown] ([unknown])" + + $ perf script -i inj --itrace=ce -F+machine_pid,+vcpu,+addr,+pid,+tid,-period --ns --time 7919.408803365,7919.408804631 -C 2 + CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803461: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803461: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803567: branches: 7f851c9b5a5a init_cacheinfo+0x3aa (/usr/lib/x86_64-linux-gnu/libc-2.31.so) => 0 [unknown] ([unknown]) + CPU 3/KVM 13376/13384 [002] 7919.408803567: branches: 0 [unknown] ([unknown]) => ffffffffc0f8ed80 vmx_vmexit+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803596: branches: ffffffffc0f6619a vmx_vcpu_run+0x26a ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803801: branches: ffffffffc0f66445 vmx_vcpu_run+0x515 ([kernel.kallsyms]) => ffffffffb2290b30 native_write_msr+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc0f661f8 vmx_vcpu_run+0x2c8 ([kernel.kallsyms]) => ffffffffc1092300 kvm_load_host_xsave_state+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc1092327 kvm_load_host_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092220 kvm_load_host_xsave_state.part.0+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662cf vmx_vcpu_run+0x39f ([kernel.kallsyms]) => ffffffffc0f63f90 vmx_recover_nmi_blocking+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662e9 vmx_vcpu_run+0x3b9 ([kernel.kallsyms]) => ffffffffc0f619a0 __vmx_complete_interrupts+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803872: branches: ffffffffc109cfb2 vcpu_enter_guest+0x752 ([kernel.kallsyms]) => ffffffffc0f5f570 vmx_handle_exit_irqoff+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803881: branches: ffffffffc109d028 vcpu_enter_guest+0x7c8 ([kernel.kallsyms]) => ffffffffb234f900 __srcu_read_lock+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc109d06f vcpu_enter_guest+0x80f ([kernel.kallsyms]) => ffffffffc0f72e30 vmx_handle_exit+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72e3d vmx_handle_exit+0xd ([kernel.kallsyms]) => ffffffffc0f727c0 __vmx_handle_exit+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72b15 __vmx_handle_exit+0x355 ([kernel.kallsyms]) => ffffffffc0f60ae0 vmx_flush_pml_buffer+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc0f72994 __vmx_handle_exit+0x1d4 ([kernel.kallsyms]) => ffffffffc10b7090 kvm_emulate_cpuid+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc10b70f1 kvm_emulate_cpuid+0x61 ([kernel.kallsyms]) => ffffffffc10b6e10 kvm_cpuid+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc10b7125 kvm_emulate_cpuid+0x95 ([kernel.kallsyms]) => ffffffffc1093110 kvm_skip_emulated_instruction+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc109311f kvm_skip_emulated_instruction+0xf ([kernel.kallsyms]) => ffffffffc0f5e180 vmx_get_rflags+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc109312a kvm_skip_emulated_instruction+0x1a ([kernel.kallsyms]) => ffffffffc0f5fd30 vmx_skip_emulated_instruction+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc0f5fd79 vmx_skip_emulated_instruction+0x49 ([kernel.kallsyms]) => ffffffffc0f5fb50 skip_emulated_instruction+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803956: branches: ffffffffc0f5fc68 skip_emulated_instruction+0x118 ([kernel.kallsyms]) => ffffffffc0f6a940 vmx_cache_reg+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803964: branches: ffffffffc0f5fc11 skip_emulated_instruction+0xc1 ([kernel.kallsyms]) => ffffffffc0f5f9e0 vmx_set_interrupt_shadow+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc109f8b1 vcpu_run+0x71 ([kernel.kallsyms]) => ffffffffc10ad2f0 kvm_cpu_has_pending_timer+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc10ad2fb kvm_cpu_has_pending_timer+0xb ([kernel.kallsyms]) => ffffffffc10b0490 apic_has_pending_timer+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803991: branches: ffffffffc109f899 vcpu_run+0x59 ([kernel.kallsyms]) => ffffffffc109c860 vcpu_enter_guest+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803993: branches: ffffffffc109cd4c vcpu_enter_guest+0x4ec ([kernel.kallsyms]) => ffffffffc0f69140 vmx_prepare_switch_to_guest+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd7d vcpu_enter_guest+0x51d ([kernel.kallsyms]) => ffffffffb234f930 __srcu_read_unlock+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd9c vcpu_enter_guest+0x53c ([kernel.kallsyms]) => ffffffffc0f609b0 vmx_sync_pir_to_irr+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc0f60a6d vmx_sync_pir_to_irr+0xbd ([kernel.kallsyms]) => ffffffffc10adc20 kvm_lapic_find_highest_irr+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804010: branches: ffffffffc0f60abd vmx_sync_pir_to_irr+0x10d ([kernel.kallsyms]) => ffffffffc0f60820 vmx_set_rvi+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804019: branches: ffffffffc109ceca vcpu_enter_guest+0x66a ([kernel.kallsyms]) => ffffffffb2249840 fpregs_assert_state_consistent+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804021: branches: ffffffffc109cf10 vcpu_enter_guest+0x6b0 ([kernel.kallsyms]) => ffffffffc0f65f30 vmx_vcpu_run+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f6603b vmx_vcpu_run+0x10b ([kernel.kallsyms]) => ffffffffb229bed0 __get_current_cr3_fast+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f66055 vmx_vcpu_run+0x125 ([kernel.kallsyms]) => ffffffffb2253050 cr4_read_shadow+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc0f6608d vmx_vcpu_run+0x15d ([kernel.kallsyms]) => ffffffffc10921e0 kvm_load_guest_xsave_state+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc1092207 kvm_load_guest_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092110 kvm_load_guest_xsave_state.part.0+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffc0f660c6 vmx_vcpu_run+0x196 ([kernel.kallsyms]) => ffffffffb22061a0 perf_guest_get_msrs+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffb22061a9 perf_guest_get_msrs+0x9 ([kernel.kallsyms]) => ffffffffb220cda0 intel_guest_get_msrs+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804039: branches: ffffffffc0f66109 vmx_vcpu_run+0x1d9 ([kernel.kallsyms]) => ffffffffc0f652c0 clear_atomic_switch_msr+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804040: branches: ffffffffc0f66119 vmx_vcpu_run+0x1e9 ([kernel.kallsyms]) => ffffffffc0f73f60 intel_pmu_lbr_is_enabled+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804042: branches: ffffffffc0f73f81 intel_pmu_lbr_is_enabled+0x21 ([kernel.kallsyms]) => ffffffffc10b68e0 kvm_find_cpuid_entry+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804045: branches: ffffffffc0f66454 vmx_vcpu_run+0x524 ([kernel.kallsyms]) => ffffffffc0f61ff0 vmx_update_hv_timer+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66142 vmx_vcpu_run+0x212 ([kernel.kallsyms]) => ffffffffc10af100 kvm_wait_lapic_expire+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66156 vmx_vcpu_run+0x226 ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66161 vmx_vcpu_run+0x231 ([kernel.kallsyms]) => ffffffffc0f8eb20 vmx_vcpu_enter_exit+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f8eb44 vmx_vcpu_enter_exit+0x24 ([kernel.kallsyms]) => ffffffffb2353e10 rcu_note_context_switch+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffb2353e1c rcu_note_context_switch+0xc ([kernel.kallsyms]) => ffffffffb2353db0 rcu_qs+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms]) + CPU 3/KVM 13376/13384 [002] 7919.408804162: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804162: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804273: branches: 7f851cb7c0e4 _dl_init+0x74 (/usr/lib/x86_64-linux-gnu/ld-2.31.so) => 7f851cb7bf50 call_init.part.0+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: 55e0c00136f0 _start+0x0 (/usr/bin/uname) => ffffffff83200ac0 asm_exc_page_fault+0x0 ([kernel.kallsyms]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: ffffffff83200ac3 asm_exc_page_fault+0x3 ([kernel.kallsyms]) => ffffffff83201290 error_entry+0x0 ([kernel.kallsyms]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804534: branches: ffffffff832012fa error_entry+0x6a ([kernel.kallsyms]) => ffffffff830b59a0 sync_regs+0x0 ([kernel.kallsyms]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff83200ad9 asm_exc_page_fault+0x19 ([kernel.kallsyms]) => ffffffff830b8210 exc_page_fault+0x0 ([kernel.kallsyms]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b82a4 exc_page_fault+0x94 ([kernel.kallsyms]) => ffffffff830b80e0 __kvm_handle_async_pf+0x0 ([kernel.kallsyms]) + VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b80ed __kvm_handle_async_pf+0xd ([kernel.kallsyms]) => ffffffff830b80c0 kvm_read_and_reset_apf_flags+0x0 ([kernel.kallsyms]) + + +Tracing Virtual Machines - Guest Code +------------------------------------- + +A common case for KVM test programs is that the test program acts as the +hypervisor, creating, running and destroying the virtual machine, and +providing the guest object code from its own object code. In this case, +the VM is not running an OS, but only the functions loaded into it by the +hypervisor test program, and conveniently, loaded at the same virtual +addresses. To support that, option "--guest-code" has been added to perf script +and perf kvm report. + +Here is an example tracing a test program from the kernel's KVM selftests: + + # perf record --kcore -e intel_pt/cyc/ -- tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 0.280 MB perf.data ] + # perf script --guest-code --itrace=bep --ns -F-period,+addr,+flags + [SNIP] + tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087733: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087836: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) + [guest/18436] 18436 [007] 10897.962087836: branches: vmentry 0 [unknown] ([unknown]) => 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962087836: branches: call 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962088248: branches: vmexit 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) + tsc_msrs_test 18436 [007] 10897.962088248: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088248: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088256: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088270: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) + [SNIP] + tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089321: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089424: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) + [guest/18436] 18436 [007] 10897.962089424: branches: vmentry 0 [unknown] ([unknown]) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jmp 40dc1b ucall+0x7b (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc39 ucall+0x99 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc37 ucall+0x97 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc50 ucall+0xb0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089878: branches: vmexit 40dc55 ucall+0xb5 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) + tsc_msrs_test 18436 [007] 10897.962089878: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089878: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089887: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089901: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) + [SNIP] + + # perf kvm --guest-code --guest --host report -i perf.data --stdio | head -20 + + # To display the perf.data header info, please use --header/--header-only options. + # + # + # Total Lost Samples: 0 + # + # Samples: 12 of event 'instructions' + # Event count (approx.): 2274583 + # + # Children Self Command Shared Object Symbol + # ........ ........ ............. .................... ........................................... + # + 54.70% 0.00% tsc_msrs_test [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe + | + ---entry_SYSCALL_64_after_hwframe + do_syscall_64 + | + |--29.44%--syscall_exit_to_user_mode + | exit_to_user_mode_prepare + | task_work_run + | __fput + + +Event Trace +----------- + +Event Trace records information about asynchronous events, for example interrupts, +faults, VM exits and entries. The information is recorded in CFE and EVD packets, +and also the Interrupt Flag is recorded on the MODE.Exec packet. The CFE packet +contains a type field to identify one of the following: + + 1 INTR interrupt, fault, exception, NMI + 2 IRET interrupt return + 3 SMI system management interrupt + 4 RSM resume from system management mode + 5 SIPI startup interprocessor interrupt + 6 INIT INIT signal + 7 VMENTRY VM-Entry + 8 VMEXIT VM-Entry + 9 VMEXIT_INTR VM-Exit due to interrupt + 10 SHUTDOWN Shutdown + +For more details, refer to the Intel 64 and IA-32 Architectures Software +Developer Manuals (version 076 or later). + +The capability to do Event Trace is indicated by the +/sys/bus/event_source/devices/intel_pt/caps/event_trace file. + +Event trace is selected for recording using the "event" config term. e.g. + + perf record -e intel_pt/event/u uname + +Event trace events are output using the --itrace I option. e.g. + + perf script --itrace=Ie + +perf script displays events containing CFE type, vector and event data, +in the form: + + evt: hw int (t) cfe: INTR IP: 1 vector: 3 PFA: 0x8877665544332211 + +The IP flag indicates if the event binds to an IP, which includes any case where +flow control packet generation is enabled, as well as when CFE packet IP bit is +set. + +perf script displays events containing changes to the Interrupt Flag in the form: + + iflag: t IFLAG: 1->0 via branch + +where "via branch" indicates a branch (interrupt or return from interrupt) and +"non branch" indicates an instruction such as CFI, STI or POPF). + +In addition, the current state of the interrupt flag is indicated by the presence +or absence of the "D" (interrupt disabled) perf script flag. If the interrupt +flag is changed, then the "t" flag is also included i.e. + + no flag, interrupts enabled IF=1 + t interrupts become disabled IF=1 -> IF=0 + D interrupts are disabled IF=0 + Dt interrupts become enabled IF=0 -> IF=1 + +The intel-pt-events.py script illustrates how to access Event Trace information +using a Python script. + + +TNT Disable +----------- + +TNT packets are disabled using the "notnt" config term. e.g. + + perf record -e intel_pt/notnt/u uname + +In that case the --itrace q option is forced because walking executable code +to reconstruct the control flow is not possible. + + +Emulated PTWRITE +---------------- + +Later perf tools support a method to emulate the ptwrite instruction, which +can be useful if hardware does not support the ptwrite instruction. + +Instead of using the ptwrite instruction, a function is used which produces +a trace that encodes the payload data into TNT packets. Here is an example +of the function: + + #include <stdint.h> + + void perf_emulate_ptwrite(uint64_t x) + __attribute__((externally_visible, noipa, no_instrument_function, naked)); + + #define PERF_EMULATE_PTWRITE_8_BITS \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" \ + "1: shl %rax\n" \ + " jc 1f\n" + + /* Undefined instruction */ + #define PERF_EMULATE_PTWRITE_UD2 ".byte 0x0f, 0x0b\n" + + #define PERF_EMULATE_PTWRITE_MAGIC PERF_EMULATE_PTWRITE_UD2 ".ascii \"perf,ptwrite \"\n" + + void perf_emulate_ptwrite(uint64_t x __attribute__ ((__unused__))) + { + /* Assumes SysV ABI : x passed in rdi */ + __asm__ volatile ( + "jmp 1f\n" + PERF_EMULATE_PTWRITE_MAGIC + "1: mov %rdi, %rax\n" + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + PERF_EMULATE_PTWRITE_8_BITS + "1: ret\n" + ); + } + +For example, a test program with the function above: + + #include <stdio.h> + #include <stdint.h> + #include <stdlib.h> + + #include "perf_emulate_ptwrite.h" + + int main(int argc, char *argv[]) + { + uint64_t x = 0; + + if (argc > 1) + x = strtoull(argv[1], NULL, 0); + perf_emulate_ptwrite(x); + return 0; + } + +Can be compiled and traced: + + $ gcc -Wall -Wextra -O3 -g -o eg_ptw eg_ptw.c + $ perf record -e intel_pt//u ./eg_ptw 0x1234567890abcdef + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 0.017 MB perf.data ] + $ perf script --itrace=ew + eg_ptw 19875 [007] 8061.235912: ptwrite: IP: 0 payload: 0x1234567890abcdef 55701249a196 perf_emulate_ptwrite+0x16 (/home/user/eg_ptw) + $ + + +EXAMPLE +------- + +Examples can be found on perf wiki page "Perf tools support for Intel® Processor Trace": + +https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace + SEE ALSO -------- |