Zephyr¶

Introduction¶

The trace recorder library is the target system component of Tracealyzer. The recorder comes pre-integrated as a configurable Zephyr module since v2.6 and is used to record a trace of RTOS events and your User Events.

An example trace can be found in the Zephyr folder.

The recorder has two main operating modes, described below.

In snapshot mode, the trace data is kept in target RAM, allowing you to take a “snapshot” at any point by saving the contents of the RAM buffer. The snapshot mode is highly optimized for memory efficiency and the resulting data rate is often just 10-20 KB/s, depending on the system. A trace buffer of a few KB is often sufficient to get a decent trace of the most recent events. This can be used in essentially any system, and even used as a “black-box” during field testing or in deployed systems.

In streaming mode, the trace data is transferred continuously to the host PC and thereby allows for tracing over long durations. This supports streaming via debug probes such as IAR I-Jet, Keil ULINKpro, and SEGGER J-Link, but may also utilize other interfaces in your system, such as USB, TCP/IP, SPI, high-speed UARTs, or device file systems.

Setting up Recorder (Zephyr v3.0.0 and Later)¶

Assuming you are running Zephyr v3.0.0 or later, the trace recorder can be enabled in your application by following these steps:

Enable Zephyr Tracing Support (CONFIG_TRACING)
Enable Percepio Tracealyzer Tracing Format (CONFIG_PERCEPIO_TRACERECORDER).
Decide on which stream port to use, i.e., how to transfer the trace data from target to host.
Select desired Recorder Start Mode.
Select desired Tracing Configuration.

If you run into problems, please refer to the Troubleshooting section.

Setting up Recorder (Zephyr v2.6.0, v2.7.0, v2.7.1)¶

Assuming you are running Zephyr v2.6.0, v2.7.0, or v2.7.1 the trace recorder can be enabled in your application by following these steps:

Enable Zephyr Tracing Support (CONFIG_TRACING).
Enable Percepio Tracealyzer Tracing Format (CONFIG_PERCEPIO_TRACERECORDER).
Select Streaming Mode (CONFIG_PERCEPIO_TRC_RECORDER_MODE_STREAMING).
Decide on which stream port to use, i.e., how to transfer the trace data from target to host.
Select desired Recorder Start Mode.
Select desired Tracing Configuration.

If you run into problems, please refer to the Troubleshooting section.

Configuration (Zephyr v3.0.0 and later)¶

The trace recorder module has several settings that allow for tuning the recorder setup. Integrated with Zephyr these settings are controlled through Zephyrs KConfig system and can be changed using menuconfig, a configuration file, or as command-line options.

Recorder Start Mode¶

CONFIG_PERCEPIO_TRC_START_MODE_START: Specifies that the tracing should start directly when the Zephyr Kernel starts.
CONFIG_PERCEPIO_TRC_START_MODE_START_AWAIT_HOST: For streaming mode only. Initializes the trace recorder and waits for a Start command from Tracealyzer before the application start.
CONFIG_PERCEPIO_TRC_START_MODE_START_FROM_HOST: For streaming mode only. Initializes the trace recorder if necessary and creates a task that waits for a Start command from Tracealyzer (“Start Recording” button). This call is not blocking.

Recorder Allocation Mode¶

Specifies how the recorder buffer is allocated (also in case of streaming, in port using the recorder’s internal temporary buffer)

CONFIG_PERCEPIO_TRC_RECORDER_BUFFER_ALLOCATION_STATIC: Static allocation (internal)
CONFIG_PERCEPIO_TRC_RECORDER_BUFFER_ALLOCATION_DYNAMIC: k_heap_alloc, requires Zephyr kernel heap and enough allocated memory.
CONFIG_PERCEPIO_TRC_RECORDER_BUFFER_ALLOCATION_CUSTOM: Use vTraceSetRecorderDataBuffer.

Control Task¶

CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_PRIORITY: In streaming mode, TzCtrl is used to receive start/stop commands from Tracealyzer and in some cases also to transmit the trace data (for stream ports that use the internal buffer, like TCP/IP). For such stream ports, make sure the TzCtrl priority is high enough to ensure reliable periodic execution and transfer of the data, but low enough to avoid disturbing any time-sensitive functions.
CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_DELAY: The delay between loops of the TzCtrl task (see CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_PRIORITY), which affects the frequency of the stack monitoring.
CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_STACK_SIZE: The stack size of the Tracealyzer Control (TzCtrl) task. See CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_PRIORITY for further information about TzCtrl.

Stack Monitor¶

CONFIG_PERCEPIO_TRC_CFG_ENABLE_STACK_MONITOR: If enabled, the recorder periodically reports the unused stack space of all active tasks. The stack monitoring runs in the Tracealyzer Control task, TzCtrl. This task is always created by the recorder when in streaming mode. In snapshot mode, the TzCtrl task is only used for stack monitoring and is not created unless this is enabled.
CONFIG_PERCEPIO_TRC_CFG_STACK_MONITOR_MAX_TASKS: Specifies This controls how many tasks can be monitored by the stack monitor. If this is too small, some tasks will be excluded and a warning is shown.
CONFIG_PERCEPIO_TRC_CFG_STACK_MONITOR_MAX_REPORTS: This defines how many tasks that will be subject to stack usage analysis for each execution of the Tracealyzer Control task (TzCtrl). Note that the stack monitoring cycles between the tasks, so this does not affect WHICH tasks are monitored, but HOW OFTEN each task stack is analyzed.

ISR Tracing¶

CONFIG_PERCEPIO_CFG_MAX_ISR_NESTING: Defines how many levels of interrupt nesting the recorder can handle, in case multiple ISRs are traced and ISR nesting is possible. If this is exceeded, the particular ISR will not be traced and the recorder then logs an error message. This setting is used to allocate an internal stack for keeping track of the previous execution context (4 bytes per entry).
CONFIG_PERCEPIO_TRC_CFG_ISR_TAILCHAINING_THRESHOLD: If tracing multiple ISRs, this setting allows for accurate display of the context-switching also in cases when the ISRs execute in direct sequence. vTraceStoreISREnd normally assumes that the ISR returns to the previous context, i.e., a task or a preempted ISR. But if another traced ISR executes in direct sequence, Tracealyzer may incorrectly display a minimal fragment of the previous context in between the ISRs. By using CONFIG_PERCEPIO_TRC_CFG_ISR_TAILCHAINING_THRESHOLD you can avoid this. This is however a threshold value that must be measured for your specific setup. See http://percepio.com/2014/03/21/isr_tailchaining_threshold/ The default setting is 0, meaning “disabled” and that you may get an extra fragments of the previous context in between tail-chained ISRs.

Streaming Config¶

CONFIG_PERCEPIO_TRC_CFG_ENTRY_SLOTS- The maximum number of symbols names that can be stored, including names for tasks and traced ISRs, other named kernel objects and user event channels.
CONFIG_PERCEPIO_TRC_CFG_ENTRY_SYMBOL_MAX_LENGTH- The maximum length of symbol names.

Streamport RTT Config¶

CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_UP_BUFFER_SIZE: Defines the size of the “up” RTT buffer (target -> host) to use for writing the trace data, for RTT buffer 1 or higher. This setting is ignored for RTT buffer 0, which can’t be reconfigured in runtime and therefore hard-coded to use the defines in SEGGER_RTT_Conf.h. Default buffer size for Tracealyzer is 5000 bytes. If you have a stand-alone J-Link probe, the can be decreased to around 1 KB. But integrated J-Link OB interfaces are slower and needs about 5-10 KB, depending on the amount of data produced.
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_DOWN_BUFFER_SIZE: Defines the size of the “down” RTT buffer (host -> target) to use for reading commands from Tracealyzer, for RTT buffer 1 or higher.
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_UP_BUFFER_INDEX: Defines the RTT buffer to use for writing the trace data. Make sure that the PC application has the same setting (File->Settings).
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_DOWN_BUFFER_INDEX: Defines the RTT buffer to use for reading the trace data. Make sure that the PC application has the same setting (File->Settings).
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_MODE This stream port for J-Link streaming relies on SEGGER RTT, that contains an internal RAM buffer read by the J-Link probes during execution.
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_MODE_BLOCK_IF_FIFO_FULL: Ensure that you get a complete and valid trace. This may however cause blocking if your streaming interface isn’t fast enough, which may disturb the real-time behavior.
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_MODE_NO_BLOCK_SKIP: In this mode, Tracealyzer will report lost events if the transfer is not fast enough. In that case, try increasing the size of the “up buffer”.

Streamport ITM Config¶

CONFIG_PERCEPIO_TRC_CFG_ITM_PORT: What ITM port to use for the ITM software events. Make sure the IDE is configured for the same channel.

Streamport Ringbuffer Config¶

CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER_SIZE: Defines the size of the ring buffer use for storing trace events.
CONFIG_PERCEPIO_TRC_STREAM_PORT_RINGBUFFER_MODE_OVERWRITE_WHEN_FULL: Configures the ring buffer to overwrite old events when full.
CONFIG_PERCEPIO_TRC_STREAM_PORT_RINGBUFFER_MODE_STOP_WHEN_FULL: Configures the ring buffer to stop accepting new events when full.

Configuration (Zephyr v2.6.0, v2.7.0, v2.7.1)¶

The trace recorder module has several settings that allow for tuning the recorder setup. Integrated with Zephyr these settings are controlled through Zephyrs KConfig system and can be changed using menuconfig, a configuration file, or as command-line options.

Recorder Mode¶

CONFIG_PERCEPIO_TRC_RECORDER_MODE_STREAMING: Sets recording mode to streaming and is mutually exclusive with CONFIG_PERCEPIO_TRC_RECORDER_MODE_SNAPSHOT= y.
CONFIG_PERCEPIO_TRC_RECORDER_MODE_SNAPSHOT: Sets recording mode to snapshot and is mutually exclusive with CONFIG_PERCEPIO_TRC_RECORDER_MODE_STREAMING= y.

Recorder Start Mode¶

CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_START: Specifies that the tracing should start directly when the Zephyr Kernel starts.
CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_START_AWAIT_HOST: For streaming mode only. Initializes the trace recorder and waits for a Start command from Tracealyzer before the application start.
CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_INIT: Initializes the trace recorder, but does not start the tracing.

Recorder Allocation Mode¶

Specifies how the recorder buffer is allocated (also in case of streaming, in port using the recorder’s internal temporary buffer)

CONFIG_PERCEPIO_TRC_RECORDER_BUFFER_ALLOCATION_STATIC: Static allocation
CONFIG_PERCEPIO_TRC_RECORDER_BUFFER_ALLOCATION_DYNAMIC: k_heap_alloc, requires Zephyr kernel heap and enough allocated memory.

Control Task¶

CONFIG_PERCEPIO_RECORDER_CFG_CTRL_TASK_PRIORITY: In streaming mode, TzCtrl is used to receive start/stop commands from Tracealyzer and in some cases also to transmit the trace data (for stream ports that use the internal buffer, like TCP/IP). For such stream ports, make sure the TzCtrl priority is high enough to ensure reliable periodic execution and transfer of the data, but low enough to avoid disturbing any time-sensitive functions.
CONFIG_PERCEPIO_RECORDER_CFG_CTRL_TASK_DELAY: The delay between loops of the TzCtrl task (see CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_PRIORITY), which affects the frequency of the stack monitoring.
CONFIG_PERCEPIO_RECORDER_CFG_CTRL_TASK_STACK_SIZE: The stack size of the Tracealyzer Control (TzCtrl) task. See CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_PRIORITY for further information about TzCtrl.
CONFIG_PERCEPIO_RECORDER_CFG_MAX_ISR_NESTING: Defines how many levels of interrupt nesting the recorder can handle, in case multiple ISRs are traced and ISR nesting is possible. If this is exceeded, the particular ISR will not be traced and the recorder then logs an error message. This setting is used to allocate an internal stack for keeping track of the previous execution context (4 bytes per entry).

Stack Monitor¶

CONFIG_PERCEPIO_RECORDER_CFG_STACK_MONITOR: If enabled, the recorder periodically reports the unused stack space of all active tasks. The stack monitoring runs in the Tracealyzer Control task, TzCtrl. This task is always created by the recorder when in streaming mode. In snapshot mode, the TzCtrl task is only used for stack monitoring and is not created unless this is enabled.
CONFIG_PERCEPIO_RECORDER_CFG_STACK_MONITOR_MAX_TASKS: Specifies This controls how many tasks can be monitored by the stack monitor. If this is too small, some tasks will be excluded and a warning is shown.
CONFIG_PERCEPIO_RECORDER_CFG_STACK_MONITOR_MAX_REPORTS: This defines how many tasks that will be subject to stack usage analysis for each execution of the Tracealyzer Control task (TzCtrl). Note that the stack monitoring cycles between the tasks, so this does not affect WHICH tasks are monitored, but HOW OFTEN each task stack is analyzed.

Streaming Config¶

CONFIG_PERCEPIO_TRC_CFG_SYMBOL_TABLE_SLOTS- The maximum number of symbols names that can be stored, including names for tasks and traced ISRs, other named kernel objects and user event channels.
CONFIG_PERCEPIO_TRC_CFG_SYMBOL_MAX_LENGTH- The maximum length of symbol names.
CONFIG_PERCEPIO_TRC_CFG_OBJECT_DATA_SLOTS- The maximum number of “object data entries”, used to store task priorities. Must be sufficient for all simultaneously active tasks, i.e., the number of created tasks minus the number of deleted/closed tasks.
CONFIG_PERCEPIO_RECORDER_TRC_STREAM_PORT_USE_INTERNAL_BUFFER Use the trace recorder for the temporary storage of trace data, periodically flushing it to the streaming interface by the trace control thread (TzCtrl).
CONFIG_PERCEPIO_TRC_CFG_PAGED_EVENT_BUFFER_PAGE_COUNT- The number of pages used by the internal trace buffer. This may need to be increased if there are missed events.
CONFIG_PERCEPIO_TRC_CFG_PAGED_EVENT_BUFFER_PAGE_SIZE- Specifies the size of each page in the internal trace buffer. This can be tuned to match any internal low-level buffers used by the streaming interface, like the Ethernet MTU (Maximum Transmission Unit).

Troubleshooting¶

Runtime errors in the recorder are often captured by the ASSERT macros. They call prvTraceError or prvTraceWarning if an internal error is detected in the recorder code, e.g., due to incorrect parameters or configuration. This attempts to save the error message in the trace and then stops the recorder. The error message is then presented when opening the trace.

But if no error message is presented, or you like to see the context of the error, it is advised to put a breakpoint on prvTraceError and/or prvTraceWarning.

You may also call xTraceGetLastError() in order to obtain any error messages in your target code. This returns NULL if no error messages are available.

In case you get “hard faults” (i.e. crashes…) when adding the recorder, this is often due to a stack overflow. The added instrumentation will increase the stack usage of your tasks somewhat, as a few function calls are added to the call stacks. If your application was close to a stack overflow before, this might push it over the edge. In that case, increase the stack size where necessary. Note that some RTOSes include stack overflow detection, although not always 100% reliable.

In case you have problems connecting to the recorder in streaming mode, or it reports lost events, make sure to check the following.

Double-check the streaming settings in Tracealyzer (File -> Settings). Also check that the stream port settings match where relevant. Using a SEGGER J-Link and RTT streaming? See the Troubleshooting J-Link RTT streaming section above. Check that tracing is properly started. Put a breakpoint on prvSetRecorderEnabled() and check if it’s reached. If using CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_INIT=y or CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_START_AWAIT_HOST, this function is only called when the “start tracing” command is received from Tracealyzer. If the breakpoint is not hit, no such start command is received from the connection, indicating a general connection problem. Note: if using CONFIG_PERCEPIO_RECORDER_CFG_START_MODE_START=y the prvSetRecorderEnabled function is called directly from within vTraceEnable. Is the trace control task (TzCtrl) executing? Put a breakpoint in the start of this task to verify this. Note that TzCtrl normally is started automatically from vTraceEnable. Is your streaming interface sufficiently fast to handle the data produced? If not, there may be lost events or blocking in the recorder, depending on your recorder settings. If blocking mode is used, this may affect the timing of your system, but ensures that data is not lost. In case of data loss, Tracealyzer will show a counter for “dropped events” in the “Connect to target system” window. Make sure the trace buffer is sufficiently large. If using J-Link streaming, the buffer size setting can be set with the CONFIG_PERCEPIO_RECORDER_TRC_CFG_RTT_BUFFER_SIZE_UP option. In most stream ports except J-Link, the trace control task (TzCtrl) performs the writing to the streaming interface. In such cases, try reducing CONFIG_PERCEPIO_RECORDER_CFG_CTRL_TASK_DELAY to make it execute more frequently. Especially if you don’t have enough RAM for a larger buffer. If your system is highly loaded, you might need to increase the priority of the trace control task to make sure it runs periodically. This is done by decreasing CONFIG_PERCEPIO_RECORDER_CFG_CTRL_TASK_PRIORITY. This is relevant for stream ports that use the internal paged buffer. Check if you get any warnings related to TRC_STREAM_PORT_WRITE_DATA or TRC_STREAM_PORT_READ_DATA. These define the I/O functions used by the stream port to read commands and write trace data, and should return zero (0) on success. Any other return values result in warnings (e.g. “TRC_STREAM_PORT_WRITE_DATA returned error (!= 0)”). This can be checked by calling xTraceGetLastError() (returns a string, or NULL if no error), or by placing a breakpoint in prvTraceWarning(). And finally, don’t hesitate to contact support@percepio.com if you need assistance.