diff options
Diffstat (limited to 'Documentation/trace')
| -rw-r--r-- | Documentation/trace/coresight/coresight-ect.rst | 222 | ||||
| -rw-r--r-- | Documentation/trace/coresight/coresight.rst | 13 | ||||
| -rw-r--r-- | Documentation/trace/events-power.rst | 21 | ||||
| -rw-r--r-- | Documentation/trace/events.rst | 63 | ||||
| -rw-r--r-- | Documentation/trace/ftrace.rst | 82 |
5 files changed, 343 insertions, 58 deletions
diff --git a/Documentation/trace/coresight/coresight-ect.rst b/Documentation/trace/coresight/coresight-ect.rst new file mode 100644 index 000000000000..ecc1e57012ef --- /dev/null +++ b/Documentation/trace/coresight/coresight-ect.rst @@ -0,0 +1,222 @@ +.. SPDX-License-Identifier: GPL-2.0 +============================================= +CoreSight Embedded Cross Trigger (CTI & CTM). +============================================= + + :Author: Mike Leach <mike.leach@linaro.org> + :Date: November 2019 + +Hardware Description +-------------------- + +The CoreSight Cross Trigger Interface (CTI) is a hardware device that takes +individual input and output hardware signals known as triggers to and from +devices and interconnects them via the Cross Trigger Matrix (CTM) to other +devices via numbered channels, in order to propagate events between devices. + +e.g.:: + + 0000000 in_trigs ::::::: + 0 C 0----------->: : +======>(other CTI channel IO) + 0 P 0<-----------: : v + 0 U 0 out_trigs : : Channels ***** ::::::: + 0000000 : CTI :<=========>*CTM*<====>: CTI :---+ + ####### in_trigs : : (id 0-3) ***** ::::::: v + # ETM #----------->: : ^ ####### + # #<-----------: : +---# ETR # + ####### out_trigs ::::::: ####### + +The CTI driver enables the programming of the CTI to attach triggers to +channels. When an input trigger becomes active, the attached channel will +become active. Any output trigger attached to that channel will also +become active. The active channel is propagated to other CTIs via the CTM, +activating connected output triggers there, unless filtered by the CTI +channel gate. + +It is also possible to activate a channel using system software directly +programming registers in the CTI. + +The CTIs are registered by the system to be associated with CPUs and/or other +CoreSight devices on the trace data path. When these devices are enabled the +attached CTIs will also be enabled. By default/on power up the CTIs have +no programmed trigger/channel attachments, so will not affect the system +until explicitly programmed. + +The hardware trigger connections between CTIs and devices is implementation +defined, unless the CPU/ETM combination is a v8 architecture, in which case +the connections have an architecturally defined standard layout. + +The hardware trigger signals can also be connected to non-CoreSight devices +(e.g. UART), or be propagated off chip as hardware IO lines. + +All the CTI devices are associated with a CTM. On many systems there will be a +single effective CTM (one CTM, or multiple CTMs all interconnected), but it is +possible that systems can have nets of CTIs+CTM that are not interconnected by +a CTM to each other. On these systems a CTM index is declared to associate +CTI devices that are interconnected via a given CTM. + +Sysfs files and directories +--------------------------- + +The CTI devices appear on the existing CoreSight bus alongside the other +CoreSight devices:: + + >$ ls /sys/bus/coresight/devices + cti_cpu0 cti_cpu2 cti_sys0 etm0 etm2 funnel0 replicator0 tmc_etr0 + cti_cpu1 cti_cpu3 cti_sys1 etm1 etm3 funnel1 tmc_etf0 tpiu0 + +The ``cti_cpu<N>`` named CTIs are associated with a CPU, and any ETM used by +that core. The ``cti_sys<N>`` CTIs are general system infrastructure CTIs that +can be associated with other CoreSight devices, or other system hardware +capable of generating or using trigger signals.:: + + >$ ls /sys/bus/coresight/devices/etm0/cti_cpu0 + channels ctmid enable nr_trigger_cons mgmt power powered regs + subsystem triggers0 triggers1 uevent + +*Key file items are:-* + * ``enable``: enables/disables the CTI. Read to determine current state. + If this shows as enabled (1), but ``powered`` shows unpowered (0), then + the enable indicates a request to enabled when the device is powered. + * ``ctmid`` : associated CTM - only relevant if system has multiple CTI+CTM + clusters that are not interconnected. + * ``nr_trigger_cons`` : total connections - triggers<N> directories. + * ``powered`` : Read to determine if the CTI is currently powered. + +*Sub-directories:-* + * ``triggers<N>``: contains list of triggers for an individual connection. + * ``channels``: Contains the channel API - CTI main programming interface. + * ``regs``: Gives access to the raw programmable CTI regs. + * ``mgmt``: the standard CoreSight management registers. + + +triggers<N> directories +~~~~~~~~~~~~~~~~~~~~~~~ + +Individual trigger connection information. This describes trigger signals for +CoreSight and non-CoreSight connections. + +Each triggers directory has a set of parameters describing the triggers for +the connection. + + * ``name`` : name of connection + * ``in_signals`` : input trigger signal indexes used in this connection. + * ``in_types`` : functional types for in signals. + * ``out_signals`` : output trigger signals for this connection. + * ``out_types`` : functional types for out signals. + +e.g:: + + >$ ls ./cti_cpu0/triggers0/ + in_signals in_types name out_signals out_types + >$ cat ./cti_cpu0/triggers0/name + cpu0 + >$ cat ./cti_cpu0/triggers0/out_signals + 0-2 + >$ cat ./cti_cpu0/triggers0/out_types + pe_edbgreq pe_dbgrestart pe_ctiirq + >$ cat ./cti_cpu0/triggers0/in_signals + 0-1 + >$ cat ./cti_cpu0/triggers0/in_types + pe_dbgtrigger pe_pmuirq + +If a connection has zero signals in either the 'in' or 'out' triggers then +those parameters will be omitted. + +Channels API Directory +~~~~~~~~~~~~~~~~~~~~~~ + +This provides an easy way to attach triggers to channels, without needing +the multiple register operations that are required if manipulating the +'regs' sub-directory elements directly. + +A number of files provide this API:: + + >$ ls ./cti_sys0/channels/ + chan_clear chan_inuse chan_xtrigs_out trigin_attach + chan_free chan_pulse chan_xtrigs_reset trigin_detach + chan_gate_disable chan_set chan_xtrigs_sel trigout_attach + chan_gate_enable chan_xtrigs_in trig_filter_enable trigout_detach + trigout_filtered + +Most access to these elements take the form:: + + echo <chan> [<trigger>] > /<device_path>/<operation> + +where the optional <trigger> is only needed for trigXX_attach | detach +operations. + +e.g.:: + + >$ echo 0 1 > ./cti_sys0/channels/trigout_attach + >$ echo 0 > ./cti_sys0/channels/chan_set + +Attaches trigout(1) to channel(0), then activates channel(0) generating a +set state on cti_sys0.trigout(1) + + +*API operations* + + * ``trigin_attach, trigout_attach``: Attach a channel to a trigger signal. + * ``trigin_detach, trigout_detach``: Detach a channel from a trigger signal. + * ``chan_set``: Set the channel - the set state will be propagated around + the CTM to other connected devices. + * ``chan_clear``: Clear the channel. + * ``chan_pulse``: Set the channel for a single CoreSight clock cycle. + * ``chan_gate_enable``: Write operation sets the CTI gate to propagate + (enable) the channel to other devices. This operation takes a channel + number. CTI gate is enabled for all channels by default at power up. Read + to list the currently enabled channels on the gate. + * ``chan_gate_disable``: Write channel number to disable gate for that + channel. + * ``chan_inuse``: Show the current channels attached to any signal + * ``chan_free``: Show channels with no attached signals. + * ``chan_xtrigs_sel``: write a channel number to select a channel to view, + read to show the selected channel number. + * ``chan_xtrigs_in``: Read to show the input triggers attached to + the selected view channel. + * ``chan_xtrigs_out``:Read to show the output triggers attached to + the selected view channel. + * ``trig_filter_enable``: Defaults to enabled, disable to allow potentially + dangerous output signals to be set. + * ``trigout_filtered``: Trigger out signals that are prevented from being + set if filtering ``trig_filter_enable`` is enabled. One use is to prevent + accidental ``EDBGREQ`` signals stopping a core. + * ``chan_xtrigs_reset``: Write 1 to clear all channel / trigger programming. + Resets device hardware to default state. + + +The example below attaches input trigger index 1 to channel 2, and output +trigger index 6 to the same channel. It then examines the state of the +channel / trigger connections using the appropriate sysfs attributes. + +The settings mean that if either input trigger 1, or channel 2 go active then +trigger out 6 will go active. We then enable the CTI, and use the software +channel control to activate channel 2. We see the active channel on the +``choutstatus`` register and the active signal on the ``trigoutstatus`` +register. Finally clearing the channel removes this. + +e.g.:: + + .../cti_sys0/channels# echo 2 1 > trigin_attach + .../cti_sys0/channels# echo 2 6 > trigout_attach + .../cti_sys0/channels# cat chan_free + 0-1,3 + .../cti_sys0/channels# cat chan_inuse + 2 + .../cti_sys0/channels# echo 2 > chan_xtrigs_sel + .../cti_sys0/channels# cat chan_xtrigs_trigin + 1 + .../cti_sys0/channels# cat chan_xtrigs_trigout + 6 + .../cti_sys0/# echo 1 > enable + .../cti_sys0/channels# echo 2 > chan_set + .../cti_sys0/channels# cat ../regs/choutstatus + 0x4 + .../cti_sys0/channels# cat ../regs/trigoutstatus + 0x40 + .../cti_sys0/channels# echo 2 > chan_clear + .../cti_sys0/channels# cat ../regs/trigoutstatus + 0x0 + .../cti_sys0/channels# cat ../regs/choutstatus + 0x0 diff --git a/Documentation/trace/coresight/coresight.rst b/Documentation/trace/coresight/coresight.rst index a566719f8e7e..108600ee1e12 100644 --- a/Documentation/trace/coresight/coresight.rst +++ b/Documentation/trace/coresight/coresight.rst @@ -491,8 +491,21 @@ interface provided for that purpose by the generic STM API:: Details on how to use the generic STM API can be found here:- :doc:`../stm` [#second]_. +The CTI & CTM Modules +--------------------- + +The CTI (Cross Trigger Interface) provides a set of trigger signals between +individual CTIs and components, and can propagate these between all CTIs via +channels on the CTM (Cross Trigger Matrix). + +A separate documentation file is provided to explain the use of these devices. +(:doc:`coresight-ect`) [#fourth]_. + + .. [#first] Documentation/ABI/testing/sysfs-bus-coresight-devices-stm .. [#second] Documentation/trace/stm.rst .. [#third] https://github.com/Linaro/perf-opencsd + +.. [#fourth] Documentation/trace/coresight/coresight-ect.rst diff --git a/Documentation/trace/events-power.rst b/Documentation/trace/events-power.rst index 2ef318962e29..f45bf11fa88d 100644 --- a/Documentation/trace/events-power.rst +++ b/Documentation/trace/events-power.rst @@ -75,16 +75,6 @@ The PM QoS events are used for QoS add/update/remove request and for target/flags update. :: - pm_qos_add_request "pm_qos_class=%s value=%d" - pm_qos_update_request "pm_qos_class=%s value=%d" - pm_qos_remove_request "pm_qos_class=%s value=%d" - pm_qos_update_request_timeout "pm_qos_class=%s value=%d, timeout_us=%ld" - -The first parameter gives the QoS class name (e.g. "CPU_DMA_LATENCY"). -The second parameter is value to be added/updated/removed. -The third parameter is timeout value in usec. -:: - pm_qos_update_target "action=%s prev_value=%d curr_value=%d" pm_qos_update_flags "action=%s prev_value=0x%x curr_value=0x%x" @@ -92,7 +82,7 @@ The first parameter gives the QoS action name (e.g. "ADD_REQ"). The second parameter is the previous QoS value. The third parameter is the current QoS value to update. -And, there are also events used for device PM QoS add/update/remove request. +There are also events used for device PM QoS add/update/remove request. :: dev_pm_qos_add_request "device=%s type=%s new_value=%d" @@ -103,3 +93,12 @@ The first parameter gives the device name which tries to add/update/remove QoS requests. The second parameter gives the request type (e.g. "DEV_PM_QOS_RESUME_LATENCY"). The third parameter is value to be added/updated/removed. + +And, there are events used for CPU latency QoS add/update/remove request. +:: + + pm_qos_add_request "value=%d" + pm_qos_update_request "value=%d" + pm_qos_remove_request "value=%d" + +The parameter is the value to be added/updated/removed. diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index ed79b220bd07..4a2ebe0bd19b 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -342,7 +342,8 @@ section of Documentation/trace/ftrace.rst), but there are major differences and the implementation isn't currently tied to it in any way, so beware about making generalizations between the two. -Note: Writing into trace_marker (See Documentation/trace/ftrace.rst) +.. Note:: + Writing into trace_marker (See Documentation/trace/ftrace.rst) can also enable triggers that are written into /sys/kernel/tracing/events/ftrace/print/trigger @@ -569,14 +570,14 @@ The first creates the event in one step, using synth_event_create(). In this method, the name of the event to create and an array defining the fields is supplied to synth_event_create(). If successful, a synthetic event with that name and fields will exist following that -call. For example, to create a new "schedtest" synthetic event: +call. For example, to create a new "schedtest" synthetic event:: ret = synth_event_create("schedtest", sched_fields, ARRAY_SIZE(sched_fields), THIS_MODULE); The sched_fields param in this example points to an array of struct synth_field_desc, each of which describes an event field by type and -name: +name:: static struct synth_field_desc sched_fields[] = { { .type = "pid_t", .name = "next_pid_field" }, @@ -615,7 +616,7 @@ synth_event_gen_cmd_array_start(), the user should create and initialize a dynevent_cmd object using synth_event_cmd_init(). For example, to create a new "schedtest" synthetic event with two -fields: +fields:: struct dynevent_cmd cmd; char *buf; @@ -631,7 +632,7 @@ fields: "u64", "ts_ns"); Alternatively, using an array of struct synth_field_desc fields -containing the same information: +containing the same information:: ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE, fields, n_fields); @@ -640,7 +641,7 @@ Once the synthetic event object has been created, it can then be populated with more fields. Fields are added one by one using synth_event_add_field(), supplying the dynevent_cmd object, a field type, and a field name. For example, to add a new int field named -"intfield", the following call should be made: +"intfield", the following call should be made:: ret = synth_event_add_field(&cmd, "int", "intfield"); @@ -649,7 +650,7 @@ the field is considered to be an array. A group of fields can also be added all at once using an array of synth_field_desc with add_synth_fields(). For example, this would add -just the first four sched_fields: +just the first four sched_fields:: ret = synth_event_add_fields(&cmd, sched_fields, 4); @@ -658,7 +659,7 @@ synth_event_add_field_str() can be used to add it as-is; it will also automatically append a ';' to the string. Once all the fields have been added, the event should be finalized and -registered by calling the synth_event_gen_cmd_end() function: +registered by calling the synth_event_gen_cmd_end() function:: ret = synth_event_gen_cmd_end(&cmd); @@ -691,7 +692,7 @@ trace array)), along with an variable number of u64 args, one for each synthetic event field, and the number of values being passed. So, to trace an event corresponding to the synthetic event definition -above, code like the following could be used: +above, code like the following could be used:: ret = synth_event_trace(create_synth_test, 7, /* number of values */ 444, /* next_pid_field */ @@ -715,7 +716,7 @@ trace array)), along with an array of u64, one for each synthetic event field. To trace an event corresponding to the synthetic event definition -above, code like the following could be used: +above, code like the following could be used:: u64 vals[7]; @@ -739,7 +740,7 @@ In order to trace a synthetic event, a pointer to the trace event file is needed. The trace_get_event_file() function can be used to get it - it will find the file in the given trace instance (in this case NULL since the top trace array is being used) while at the same time -preventing the instance containing it from going away: +preventing the instance containing it from going away:: schedtest_event_file = trace_get_event_file(NULL, "synthetic", "schedtest"); @@ -751,31 +752,31 @@ To enable a synthetic event from the kernel, trace_array_set_clr_event() can be used (which is not specific to synthetic events, so does need the "synthetic" system name to be specified explicitly). -To enable the event, pass 'true' to it: +To enable the event, pass 'true' to it:: trace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", true); -To disable it pass false: +To disable it pass false:: trace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false); Finally, synth_event_trace_array() can be used to actually trace the -event, which should be visible in the trace buffer afterwards: +event, which should be visible in the trace buffer afterwards:: ret = synth_event_trace_array(schedtest_event_file, vals, ARRAY_SIZE(vals)); To remove the synthetic event, the event should be disabled, and the -trace instance should be 'put' back using trace_put_event_file(): +trace instance should be 'put' back using trace_put_event_file():: trace_array_set_clr_event(schedtest_event_file->tr, "synthetic", "schedtest", false); trace_put_event_file(schedtest_event_file); If those have been successful, synth_event_delete() can be called to -remove the event: +remove the event:: ret = synth_event_delete("schedtest"); @@ -784,7 +785,7 @@ remove the event: To trace a synthetic using the piecewise method described above, the synth_event_trace_start() function is used to 'open' the synthetic -event trace: +event trace:: struct synth_trace_state trace_state; @@ -809,7 +810,7 @@ along with the value to set the next field in the event. After each field is set, the 'cursor' points to the next field, which will be set by the subsequent call, continuing until all the fields have been set in order. The same sequence of calls as in the above examples using -this method would be (without error-handling code): +this method would be (without error-handling code):: /* next_pid_field */ ret = synth_event_add_next_val(777, &trace_state); @@ -837,7 +838,7 @@ used. Each call is passed the same synth_trace_state object used in the synth_event_trace_start(), along with the field name of the field to set and the value to set it to. The same sequence of calls as in the above examples using this method would be (without error-handling -code): +code):: ret = synth_event_add_val("next_pid_field", 777, &trace_state); ret = synth_event_add_val("next_comm_field", (u64)"silly putty", @@ -855,7 +856,7 @@ can be used but not both at the same time. Finally, the event won't be actually traced until it's 'closed', which is done using synth_event_trace_end(), which takes only the -struct synth_trace_state object used in the previous calls: +struct synth_trace_state object used in the previous calls:: ret = synth_event_trace_end(&trace_state); @@ -878,7 +879,7 @@ function. Before calling kprobe_event_gen_cmd_start(), the user should create and initialize a dynevent_cmd object using kprobe_event_cmd_init(). -For example, to create a new "schedtest" kprobe event with two fields: +For example, to create a new "schedtest" kprobe event with two fields:: struct dynevent_cmd cmd; char *buf; @@ -900,18 +901,18 @@ Once the kprobe event object has been created, it can then be populated with more fields. Fields can be added using kprobe_event_add_fields(), supplying the dynevent_cmd object along with a variable arg list of probe fields. For example, to add a -couple additional fields, the following call could be made: +couple additional fields, the following call could be made:: ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)"); Once all the fields have been added, the event should be finalized and registered by calling the kprobe_event_gen_cmd_end() or kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe -or kretprobe command was started: +or kretprobe command was started:: ret = kprobe_event_gen_cmd_end(&cmd); -or +or:: ret = kretprobe_event_gen_cmd_end(&cmd); @@ -920,13 +921,13 @@ events. Similarly, a kretprobe event can be created using kretprobe_event_gen_cmd_start() with a probe name and location and -additional params such as $retval: +additional params such as $retval:: ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test", "do_sys_open", "$retval"); Similar to the synthetic event case, code like the following can be -used to enable the newly created kprobe event: +used to enable the newly created kprobe event:: gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); @@ -934,7 +935,7 @@ used to enable the newly created kprobe event: "kprobes", "gen_kprobe_test", true); Finally, also similar to synthetic events, the following code can be -used to give the kprobe event file back and delete the event: +used to give the kprobe event file back and delete the event:: trace_put_event_file(gen_kprobe_test); @@ -963,7 +964,7 @@ are described below. The first step in building a new command string is to create and initialize an instance of a dynevent_cmd. Here, for instance, we -create a dynevent_cmd on the stack and initialize it: +create a dynevent_cmd on the stack and initialize it:: struct dynevent_cmd cmd; char *buf; @@ -989,7 +990,7 @@ calls to argument-adding functions. To add a single argument, define and initialize a struct dynevent_arg or struct dynevent_arg_pair object. Here's an example of the simplest possible arg addition, which is simply to append the given string as -a whitespace-separated argument to the command: +a whitespace-separated argument to the command:: struct dynevent_arg arg; @@ -1007,7 +1008,7 @@ the arg. Here's another more complicated example using an 'arg pair', which is used to create an argument that consists of a couple components added together as a unit, for example, a 'type field_name;' arg or a simple -expression arg e.g. 'flags=%cx': +expression arg e.g. 'flags=%cx':: struct dynevent_arg_pair arg_pair; @@ -1031,7 +1032,7 @@ Any number of dynevent_*_add() calls can be made to build up the string (until its length surpasses cmd->maxlen). When all the arguments have been added and the command string is complete, the only thing left to do is run the command, which happens by simply calling -dynevent_create(): +dynevent_create():: ret = dynevent_create(&cmd); diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst index ff658e27d25b..3b5614b1d1a5 100644 --- a/Documentation/trace/ftrace.rst +++ b/Documentation/trace/ftrace.rst @@ -125,10 +125,13 @@ of ftrace. Here is a list of some of the key files: trace: This file holds the output of the trace in a human - readable format (described below). Note, tracing is temporarily - disabled when the file is open for reading. Once all readers - are closed, tracing is re-enabled. Opening this file for + readable format (described below). Opening this file for writing with the O_TRUNC flag clears the ring buffer content. + Note, this file is not a consumer. If tracing is off + (no tracer running, or tracing_on is zero), it will produce + the same output each time it is read. When tracing is on, + it may produce inconsistent results as it tries to read + the entire buffer without consuming it. trace_pipe: @@ -142,9 +145,7 @@ of ftrace. Here is a list of some of the key files: will not be read again with a sequential read. The "trace" file is static, and if the tracer is not adding more data, it will display the same - information every time it is read. Unlike the - "trace" file, opening this file for reading will not - temporarily disable tracing. + information every time it is read. trace_options: @@ -262,6 +263,20 @@ of ftrace. Here is a list of some of the key files: traced by the function tracer as well. This option will also cause PIDs of tasks that exit to be removed from the file. + set_ftrace_notrace_pid: + + Have the function tracer ignore threads whose PID are listed in + this file. + + If the "function-fork" option is set, then when a task whose + PID is listed in this file forks, the child's PID will + automatically be added to this file, and the child will not be + traced by the function tracer as well. This option will also + cause PIDs of tasks that exit to be removed from the file. + + If a PID is in both this file and "set_ftrace_pid", then this + file takes precedence, and the thread will not be traced. + set_event_pid: Have the events only trace a task with a PID listed in this file. @@ -273,6 +288,19 @@ of ftrace. Here is a list of some of the key files: cause the PIDs of tasks to be removed from this file when the task exits. + set_event_notrace_pid: + + Have the events not trace a task with a PID listed in this file. + Note, sched_switch and sched_wakeup will trace threads not listed + in this file, even if a thread's PID is in the file if the + sched_switch or sched_wakeup events also trace a thread that should + be traced. + + To have the PIDs of children of tasks with their PID in this file + added on fork, enable the "event-fork" option. That option will also + cause the PIDs of tasks to be removed from this file when the task + exits. + set_graph_function: Functions listed in this file will cause the function graph @@ -1125,6 +1153,12 @@ Here are the available options: the trace displays additional information about the latency, as described in "Latency trace format". + pause-on-trace + When set, opening the trace file for read, will pause + writing to the ring buffer (as if tracing_on was set to zero). + This simulates the original behavior of the trace file. + When the file is closed, tracing will be enabled again. + record-cmd When any event or tracer is enabled, a hook is enabled in the sched_switch trace point to fill comm cache @@ -1176,6 +1210,8 @@ Here are the available options: tasks fork. Also, when tasks with PIDs in set_event_pid exit, their PIDs will be removed from the file. + This affects PIDs listed in set_event_notrace_pid as well. + function-trace The latency tracers will enable function tracing if this option is enabled (default it is). When @@ -1190,6 +1226,8 @@ Here are the available options: set_ftrace_pid exit, their PIDs will be removed from the file. + This affects PIDs in set_ftrace_notrace_pid as well. + display-graph When set, the latency tracers (irqsoff, wakeup, etc) will use function graph tracing instead of function tracing. @@ -2126,6 +2164,8 @@ periodically make a CPU constantly busy with interrupts disabled. # cat trace # tracer: hwlat # + # entries-in-buffer/entries-written: 13/13 #P:8 + # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq @@ -2133,12 +2173,18 @@ periodically make a CPU constantly busy with interrupts disabled. # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | - <...>-3638 [001] d... 19452.055471: #1 inner/outer(us): 12/14 ts:1499801089.066141940 - <...>-3638 [003] d... 19454.071354: #2 inner/outer(us): 11/9 ts:1499801091.082164365 - <...>-3638 [002] dn.. 19461.126852: #3 inner/outer(us): 12/9 ts:1499801098.138150062 - <...>-3638 [001] d... 19488.340960: #4 inner/outer(us): 8/12 ts:1499801125.354139633 - <...>-3638 [003] d... 19494.388553: #5 inner/outer(us): 8/12 ts:1499801131.402150961 - <...>-3638 [003] d... 19501.283419: #6 inner/outer(us): 0/12 ts:1499801138.297435289 nmi-total:4 nmi-count:1 + <...>-1729 [001] d... 678.473449: #1 inner/outer(us): 11/12 ts:1581527483.343962693 count:6 + <...>-1729 [004] d... 689.556542: #2 inner/outer(us): 16/9 ts:1581527494.889008092 count:1 + <...>-1729 [005] d... 714.756290: #3 inner/outer(us): 16/16 ts:1581527519.678961629 count:5 + <...>-1729 [001] d... 718.788247: #4 inner/outer(us): 9/17 ts:1581527523.889012713 count:1 + <...>-1729 [002] d... 719.796341: #5 inner/outer(us): 13/9 ts:1581527524.912872606 count:1 + <...>-1729 [006] d... 844.787091: #6 inner/outer(us): 9/12 ts:1581527649.889048502 count:2 + <...>-1729 [003] d... 849.827033: #7 inner/outer(us): 18/9 ts:1581527654.889013793 count:1 + <...>-1729 [007] d... 853.859002: #8 inner/outer(us): 9/12 ts:1581527658.889065736 count:1 + <...>-1729 [001] d... 855.874978: #9 inner/outer(us): 9/11 ts:1581527660.861991877 count:1 + <...>-1729 [001] d... 863.938932: #10 inner/outer(us): 9/11 ts:1581527668.970010500 count:1 nmi-total:7 nmi-count:1 + <...>-1729 [007] d... 878.050780: #11 inner/outer(us): 9/12 ts:1581527683.385002600 count:1 nmi-total:5 nmi-count:1 + <...>-1729 [007] d... 886.114702: #12 inner/outer(us): 9/12 ts:1581527691.385001600 count:1 The above output is somewhat the same in the header. All events will have @@ -2148,7 +2194,7 @@ interrupts disabled 'd'. Under the FUNCTION title there is: This is the count of events recorded that were greater than the tracing_threshold (See below). - inner/outer(us): 12/14 + inner/outer(us): 11/11 This shows two numbers as "inner latency" and "outer latency". The test runs in a loop checking a timestamp twice. The latency detected within @@ -2156,11 +2202,15 @@ interrupts disabled 'd'. Under the FUNCTION title there is: after the previous timestamp and the next timestamp in the loop is the "outer latency". - ts:1499801089.066141940 + ts:1581527483.343962693 + + The absolute timestamp that the first latency was recorded in the window. + + count:6 - The absolute timestamp that the event happened. + The number of times a latency was detected during the window. - nmi-total:4 nmi-count:1 + nmi-total:7 nmi-count:1 On architectures that support it, if an NMI comes in during the test, the time spent in NMI is reported in "nmi-total" (in |