| Coresight - HW Assisted Tracing on ARM |
| ====================================== |
| |
| Author: Mathieu Poirier <mathieu.poirier@linaro.org> |
| Date: September 11th, 2014 |
| |
| Introduction |
| ------------ |
| |
| Coresight is an umbrella of technologies allowing for the debugging of ARM |
| based SoC. It includes solutions for JTAG and HW assisted tracing. This |
| document is concerned with the latter. |
| |
| HW assisted tracing is becoming increasingly useful when dealing with systems |
| that have many SoCs and other components like GPU and DMA engines. ARM has |
| developed a HW assisted tracing solution by means of different components, each |
| being added to a design at synthesis time to cater to specific tracing needs. |
| Components are generally categorised as source, link and sinks and are |
| (usually) discovered using the AMBA bus. |
| |
| "Sources" generate a compressed stream representing the processor instruction |
| path based on tracing scenarios as configured by users. From there the stream |
| flows through the coresight system (via ATB bus) using links that are connecting |
| the emanating source to a sink(s). Sinks serve as endpoints to the coresight |
| implementation, either storing the compressed stream in a memory buffer or |
| creating an interface to the outside world where data can be transferred to a |
| host without fear of filling up the onboard coresight memory buffer. |
| |
| At typical coresight system would look like this: |
| |
| ***************************************************************** |
| **************************** AMBA AXI ****************************===|| |
| ***************************************************************** || |
| ^ ^ | || |
| | | * ** |
| 0000000 ::::: 0000000 ::::: ::::: @@@@@@@ |||||||||||| |
| 0 CPU 0<-->: C : 0 CPU 0<-->: C : : C : @ STM @ || System || |
| |->0000000 : T : |->0000000 : T : : T :<--->@@@@@ || Memory || |
| | #######<-->: I : | #######<-->: I : : I : @@@<-| |||||||||||| |
| | # ETM # ::::: | # PTM # ::::: ::::: @ | |
| | ##### ^ ^ | ##### ^ ! ^ ! . | ||||||||| |
| | |->### | ! | |->### | ! | ! . | || DAP || |
| | | # | ! | | # | ! | ! . | ||||||||| |
| | | . | ! | | . | ! | ! . | | | |
| | | . | ! | | . | ! | ! . | | * |
| | | . | ! | | . | ! | ! . | | SWD/ |
| | | . | ! | | . | ! | ! . | | JTAG |
| *****************************************************************<-| |
| *************************** AMBA Debug APB ************************ |
| ***************************************************************** |
| | . ! . ! ! . | |
| | . * . * * . | |
| ***************************************************************** |
| ******************** Cross Trigger Matrix (CTM) ******************* |
| ***************************************************************** |
| | . ^ . . | |
| | * ! * * | |
| ***************************************************************** |
| ****************** AMBA Advanced Trace Bus (ATB) ****************** |
| ***************************************************************** |
| | ! =============== | |
| | * ===== F =====<---------| |
| | ::::::::: ==== U ==== |
| |-->:: CTI ::<!! === N === |
| | ::::::::: ! == N == |
| | ^ * == E == |
| | ! &&&&&&&&& IIIIIII == L == |
| |------>&& ETB &&<......II I ======= |
| | ! &&&&&&&&& II I . |
| | ! I I . |
| | ! I REP I<.......... |
| | ! I I |
| | !!>&&&&&&&&& II I *Source: ARM ltd. |
| |------>& TPIU &<......II I DAP = Debug Access Port |
| &&&&&&&&& IIIIIII ETM = Embedded Trace Macrocell |
| ; PTM = Program Trace Macrocell |
| ; CTI = Cross Trigger Interface |
| * ETB = Embedded Trace Buffer |
| To trace port TPIU= Trace Port Interface Unit |
| SWD = Serial Wire Debug |
| |
| While on target configuration of the components is done via the APB bus, |
| all trace data are carried out-of-band on the ATB bus. The CTM provides |
| a way to aggregate and distribute signals between CoreSight components. |
| |
| The coresight framework provides a central point to represent, configure and |
| manage coresight devices on a platform. This first implementation centers on |
| the basic tracing functionality, enabling components such ETM/PTM, funnel, |
| replicator, TMC, TPIU and ETB. Future work will enable more |
| intricate IP blocks such as STM and CTI. |
| |
| |
| Acronyms and Classification |
| --------------------------- |
| |
| Acronyms: |
| |
| PTM: Program Trace Macrocell |
| ETM: Embedded Trace Macrocell |
| STM: System trace Macrocell |
| ETB: Embedded Trace Buffer |
| ITM: Instrumentation Trace Macrocell |
| TPIU: Trace Port Interface Unit |
| TMC-ETR: Trace Memory Controller, configured as Embedded Trace Router |
| TMC-ETF: Trace Memory Controller, configured as Embedded Trace FIFO |
| CTI: Cross Trigger Interface |
| |
| Classification: |
| |
| Source: |
| ETMv3.x ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM |
| Link: |
| Funnel, replicator (intelligent or not), TMC-ETR |
| Sinks: |
| ETBv1.0, ETB1.1, TPIU, TMC-ETF |
| Misc: |
| CTI |
| |
| |
| Device Tree Bindings |
| ---------------------- |
| |
| See Documentation/devicetree/bindings/arm/coresight.txt for details. |
| |
| As of this writing drivers for ITM, STMs and CTIs are not provided but are |
| expected to be added as the solution matures. |
| |
| |
| Framework and implementation |
| ---------------------------- |
| |
| The coresight framework provides a central point to represent, configure and |
| manage coresight devices on a platform. Any coresight compliant device can |
| register with the framework for as long as they use the right APIs: |
| |
| struct coresight_device *coresight_register(struct coresight_desc *desc); |
| void coresight_unregister(struct coresight_device *csdev); |
| |
| The registering function is taking a "struct coresight_device *csdev" and |
| register the device with the core framework. The unregister function takes |
| a reference to a "struct coresight_device", obtained at registration time. |
| |
| If everything goes well during the registration process the new devices will |
| show up under /sys/bus/coresight/devices, as showns here for a TC2 platform: |
| |
| root:~# ls /sys/bus/coresight/devices/ |
| replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm |
| 20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm |
| root:~# |
| |
| The functions take a "struct coresight_device", which looks like this: |
| |
| struct coresight_desc { |
| enum coresight_dev_type type; |
| struct coresight_dev_subtype subtype; |
| const struct coresight_ops *ops; |
| struct coresight_platform_data *pdata; |
| struct device *dev; |
| const struct attribute_group **groups; |
| }; |
| |
| |
| The "coresight_dev_type" identifies what the device is, i.e, source link or |
| sink while the "coresight_dev_subtype" will characterise that type further. |
| |
| The "struct coresight_ops" is mandatory and will tell the framework how to |
| perform base operations related to the components, each component having |
| a different set of requirement. For that "struct coresight_ops_sink", |
| "struct coresight_ops_link" and "struct coresight_ops_source" have been |
| provided. |
| |
| The next field, "struct coresight_platform_data *pdata" is acquired by calling |
| "of_get_coresight_platform_data()", as part of the driver's _probe routine and |
| "struct device *dev" gets the device reference embedded in the "amba_device": |
| |
| static int etm_probe(struct amba_device *adev, const struct amba_id *id) |
| { |
| ... |
| ... |
| drvdata->dev = &adev->dev; |
| ... |
| } |
| |
| Specific class of device (source, link, or sink) have generic operations |
| that can be performed on them (see "struct coresight_ops"). The |
| "**groups" is a list of sysfs entries pertaining to operations |
| specific to that component only. "Implementation defined" customisations are |
| expected to be accessed and controlled using those entries. |
| |
| |
| Device Naming scheme |
| ------------------------ |
| The devices that appear on the "coresight" bus were named the same as their |
| parent devices, i.e, the real devices that appears on AMBA bus or the platform bus. |
| Thus the names were based on the Linux Open Firmware layer naming convention, |
| which follows the base physical address of the device followed by the device |
| type. e.g: |
| |
| root:~# ls /sys/bus/coresight/devices/ |
| 20010000.etf 20040000.funnel 20100000.stm 22040000.etm |
| 22140000.etm 230c0000.funnel 23240000.etm 20030000.tpiu |
| 20070000.etr 20120000.replicator 220c0000.funnel |
| 23040000.etm 23140000.etm 23340000.etm |
| |
| However, with the introduction of ACPI support, the names of the real |
| devices are a bit cryptic and non-obvious. Thus, a new naming scheme was |
| introduced to use more generic names based on the type of the device. The |
| following rules apply: |
| |
| 1) Devices that are bound to CPUs, are named based on the CPU logical |
| number. |
| |
| e.g, ETM bound to CPU0 is named "etm0" |
| |
| 2) All other devices follow a pattern, "<device_type_prefix>N", where : |
| |
| <device_type_prefix> - A prefix specific to the type of the device |
| N - a sequential number assigned based on the order |
| of probing. |
| |
| e.g, tmc_etf0, tmc_etr0, funnel0, funnel1 |
| |
| Thus, with the new scheme the devices could appear as : |
| |
| root:~# ls /sys/bus/coresight/devices/ |
| etm0 etm1 etm2 etm3 etm4 etm5 funnel0 |
| funnel1 funnel2 replicator0 stm0 tmc_etf0 tmc_etr0 tpiu0 |
| |
| Some of the examples below might refer to old naming scheme and some |
| to the newer scheme, to give a confirmation that what you see on your |
| system is not unexpected. One must use the "names" as they appear on |
| the system under specified locations. |
| |
| How to use the tracer modules |
| ----------------------------- |
| |
| There are two ways to use the Coresight framework: 1) using the perf cmd line |
| tools and 2) interacting directly with the Coresight devices using the sysFS |
| interface. Preference is given to the former as using the sysFS interface |
| requires a deep understanding of the Coresight HW. The following sections |
| provide details on using both methods. |
| |
| 1) Using the sysFS interface: |
| |
| Before trace collection can start, a coresight sink needs to be identified. |
| There is no limit on the amount of sinks (nor sources) that can be enabled at |
| any given moment. As a generic operation, all device pertaining to the sink |
| class will have an "active" entry in sysfs: |
| |
| root:/sys/bus/coresight/devices# ls |
| replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm |
| 20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm |
| root:/sys/bus/coresight/devices# ls 20010000.etb |
| enable_sink status trigger_cntr |
| root:/sys/bus/coresight/devices# echo 1 > 20010000.etb/enable_sink |
| root:/sys/bus/coresight/devices# cat 20010000.etb/enable_sink |
| 1 |
| root:/sys/bus/coresight/devices# |
| |
| At boot time the current etm3x driver will configure the first address |
| comparator with "_stext" and "_etext", essentially tracing any instruction |
| that falls within that range. As such "enabling" a source will immediately |
| trigger a trace capture: |
| |
| root:/sys/bus/coresight/devices# echo 1 > 2201c000.ptm/enable_source |
| root:/sys/bus/coresight/devices# cat 2201c000.ptm/enable_source |
| 1 |
| root:/sys/bus/coresight/devices# cat 20010000.etb/status |
| Depth: 0x2000 |
| Status: 0x1 |
| RAM read ptr: 0x0 |
| RAM wrt ptr: 0x19d3 <----- The write pointer is moving |
| Trigger cnt: 0x0 |
| Control: 0x1 |
| Flush status: 0x0 |
| Flush ctrl: 0x2001 |
| root:/sys/bus/coresight/devices# |
| |
| Trace collection is stopped the same way: |
| |
| root:/sys/bus/coresight/devices# echo 0 > 2201c000.ptm/enable_source |
| root:/sys/bus/coresight/devices# |
| |
| The content of the ETB buffer can be harvested directly from /dev: |
| |
| root:/sys/bus/coresight/devices# dd if=/dev/20010000.etb \ |
| of=~/cstrace.bin |
| |
| 64+0 records in |
| 64+0 records out |
| 32768 bytes (33 kB) copied, 0.00125258 s, 26.2 MB/s |
| root:/sys/bus/coresight/devices# |
| |
| The file cstrace.bin can be decompressed using "ptm2human", DS-5 or Trace32. |
| |
| Following is a DS-5 output of an experimental loop that increments a variable up |
| to a certain value. The example is simple and yet provides a glimpse of the |
| wealth of possibilities that coresight provides. |
| |
| Info Tracing enabled |
| Instruction 106378866 0x8026B53C E52DE004 false PUSH {lr} |
| Instruction 0 0x8026B540 E24DD00C false SUB sp,sp,#0xc |
| Instruction 0 0x8026B544 E3A03000 false MOV r3,#0 |
| Instruction 0 0x8026B548 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Timestamp Timestamp: 17106715833 |
| Instruction 319 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Instruction 9 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Instruction 10 0x8026B54C E59D3004 false LDR r3,[sp,#4] |
| Instruction 0 0x8026B550 E3530004 false CMP r3,#4 |
| Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1 |
| Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4] |
| Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c |
| Instruction 6 0x8026B560 EE1D3F30 false MRC p15,#0x0,r3,c13,c0,#1 |
| Instruction 0 0x8026B564 E1A0100D false MOV r1,sp |
| Instruction 0 0x8026B568 E3C12D7F false BIC r2,r1,#0x1fc0 |
| Instruction 0 0x8026B56C E3C2203F false BIC r2,r2,#0x3f |
| Instruction 0 0x8026B570 E59D1004 false LDR r1,[sp,#4] |
| Instruction 0 0x8026B574 E59F0010 false LDR r0,[pc,#16] ; [0x8026B58C] = 0x80550368 |
| Instruction 0 0x8026B578 E592200C false LDR r2,[r2,#0xc] |
| Instruction 0 0x8026B57C E59221D0 false LDR r2,[r2,#0x1d0] |
| Instruction 0 0x8026B580 EB07A4CF true BL {pc}+0x1e9344 ; 0x804548c4 |
| Info Tracing enabled |
| Instruction 13570831 0x8026B584 E28DD00C false ADD sp,sp,#0xc |
| Instruction 0 0x8026B588 E8BD8000 true LDM sp!,{pc} |
| Timestamp Timestamp: 17107041535 |
| |
| 2) Using perf framework: |
| |
| Coresight tracers are represented using the Perf framework's Performance |
| Monitoring Unit (PMU) abstraction. As such the perf framework takes charge of |
| controlling when tracing gets enabled based on when the process of interest is |
| scheduled. When configured in a system, Coresight PMUs will be listed when |
| queried by the perf command line tool: |
| |
| linaro@linaro-nano:~$ ./perf list pmu |
| |
| List of pre-defined events (to be used in -e): |
| |
| cs_etm// [Kernel PMU event] |
| |
| linaro@linaro-nano:~$ |
| |
| Regardless of the number of tracers available in a system (usually equal to the |
| amount of processor cores), the "cs_etm" PMU will be listed only once. |
| |
| A Coresight PMU works the same way as any other PMU, i.e the name of the PMU is |
| listed along with configuration options within forward slashes '/'. Since a |
| Coresight system will typically have more than one sink, the name of the sink to |
| work with needs to be specified as an event option. |
| On newer kernels the available sinks are listed in sysFS under: |
| ($SYSFS)/bus/event_source/devices/cs_etm/sinks/ |
| |
| root@localhost:/sys/bus/event_source/devices/cs_etm/sinks# ls |
| tmc_etf0 tmc_etr0 tpiu0 |
| |
| On older kernels, this may need to be found from the list of coresight devices, |
| available under ($SYSFS)/bus/coresight/devices/: |
| |
| root:~# ls /sys/bus/coresight/devices/ |
| etm0 etm1 etm2 etm3 etm4 etm5 funnel0 |
| funnel1 funnel2 replicator0 stm0 tmc_etf0 tmc_etr0 tpiu0 |
| |
| root@linaro-nano:~# perf record -e cs_etm/@tmc_etr0/u --per-thread program |
| |
| As mentioned above in section "Device Naming scheme", the names of the devices could |
| look different from what is used in the example above. One must use the device names |
| as it appears under the sysFS. |
| |
| The syntax within the forward slashes '/' is important. The '@' character |
| tells the parser that a sink is about to be specified and that this is the sink |
| to use for the trace session. |
| |
| More information on the above and other example on how to use Coresight with |
| the perf tools can be found in the "HOWTO.md" file of the openCSD gitHub |
| repository [3]. |
| |
| 2.1) AutoFDO analysis using the perf tools: |
| |
| perf can be used to record and analyze trace of programs. |
| |
| Execution can be recorded using 'perf record' with the cs_etm event, |
| specifying the name of the sink to record to, e.g: |
| |
| perf record -e cs_etm/@tmc_etr0/u --per-thread |
| |
| The 'perf report' and 'perf script' commands can be used to analyze execution, |
| synthesizing instruction and branch events from the instruction trace. |
| 'perf inject' can be used to replace the trace data with the synthesized events. |
| The --itrace option controls the type and frequency of synthesized events |
| (see perf documentation). |
| |
| Note that only 64-bit programs are currently supported - further work is |
| required to support instruction decode of 32-bit Arm programs. |
| |
| |
| Generating coverage files for Feedback Directed Optimization: AutoFDO |
| --------------------------------------------------------------------- |
| |
| 'perf inject' accepts the --itrace option in which case tracing data is |
| removed and replaced with the synthesized events. e.g. |
| |
| perf inject --itrace --strip -i perf.data -o perf.data.new |
| |
| Below is an example of using ARM ETM for autoFDO. It requires autofdo |
| (https://github.com/google/autofdo) and gcc version 5. The bubble |
| sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial). |
| |
| $ gcc-5 -O3 sort.c -o sort |
| $ taskset -c 2 ./sort |
| Bubble sorting array of 30000 elements |
| 5910 ms |
| |
| $ perf record -e cs_etm/@tmc_etr0/u --per-thread taskset -c 2 ./sort |
| Bubble sorting array of 30000 elements |
| 12543 ms |
| [ perf record: Woken up 35 times to write data ] |
| [ perf record: Captured and wrote 69.640 MB perf.data ] |
| |
| $ perf inject -i perf.data -o inj.data --itrace=il64 --strip |
| $ create_gcov --binary=./sort --profile=inj.data --gcov=sort.gcov -gcov_version=1 |
| $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo |
| $ taskset -c 2 ./sort_autofdo |
| Bubble sorting array of 30000 elements |
| 5806 ms |
| |
| |
| How to use the STM module |
| ------------------------- |
| |
| Using the System Trace Macrocell module is the same as the tracers - the only |
| difference is that clients are driving the trace capture rather |
| than the program flow through the code. |
| |
| As with any other CoreSight component, specifics about the STM tracer can be |
| found in sysfs with more information on each entry being found in [1]: |
| |
| root@genericarmv8:~# ls /sys/bus/coresight/devices/stm0 |
| enable_source hwevent_select port_enable subsystem uevent |
| hwevent_enable mgmt port_select traceid |
| root@genericarmv8:~# |
| |
| Like any other source a sink needs to be identified and the STM enabled before |
| being used: |
| |
| root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/tmc_etf0/enable_sink |
| root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/stm0/enable_source |
| |
| From there user space applications can request and use channels using the devfs |
| interface provided for that purpose by the generic STM API: |
| |
| root@genericarmv8:~# ls -l /dev/stm0 |
| crw------- 1 root root 10, 61 Jan 3 18:11 /dev/stm0 |
| root@genericarmv8:~# |
| |
| Details on how to use the generic STM API can be found here [2]. |
| |
| [1]. Documentation/ABI/testing/sysfs-bus-coresight-devices-stm |
| [2]. Documentation/trace/stm.rst |
| [3]. https://github.com/Linaro/perf-opencsd |
