Tracealyzer for Zephyr

Solve Issues Quickly

• Visualize task execution
• Spot issues and drill down
• Full system debugging

Measure Performance

• CPU load, timing variations
• Analyze memory usage
• Find inefficient code

Tracealyzer provides many views for different perspectives of your runtime system. All view are connected for a streamlined workflow. Spot anomalies in high-level overviews and then drill down into the details to see the cause.

The trace view provides a detailed timeline of task scheduling and interrupts, kernel API calls as well as custom “user event” logging in the application code. This is helps you debug your code and lets you verify that your code executes as intended. You can also learn how to improve the software design for better performance.

The large set of visual overviews includes for example CPU load, task execution times, stack usage and heap memory allocation. Such overviews provides a profile of the resource usage over time, which helps you see the big picture, spot anomalies and optimize the system.

The Zephyr kernel provides multithreading, where each task has its own stack and a fixed scheduling priority which is very important for functional correctness and performance. Tracealyzer lets you analyze the behavior and performance of different priority assignments, as well as the stack usage of the tasks. If your stacks are too small, you risk nasty bugs due to stack overflow. If they are too large, you are wasting precious RAM that might be needed elsewhere in your application.

The trace view shows both the task scheduling and calls to kernel API functions. This allows you to see exactly when tasks are activated, when they actually execute, and why they sometimes don’t execute as intended. You can also see an overview showing which tasks are consuming the processor time, as shown in the “CPU Load Graph”. Detailed statistics is also available, like task execution times and response times.

Queue Events

Queues allow tasks to communicate by sending and receiving messages. Queues have a fixed number of slots and messages are normally buffered in FIFO order.

Queue operations might block if trying to send a message to a full queue, or trying to read a message from an empty queue. The screenshot shows how Tracealyzer displays queue usage and blocking. The TX task sends two messages to the queue. This wakes up the RX task (hence the green label) and it receives the two messages. Finally, the task is blocked (red label) when trying to read from the queue since now empty.

The queue forms a dependency between TX and RX, that can be seen in the Communication Flow graph below. The direction of the arrows show the usage, i.e. that TX is sending to the queue and RX is receiving from it. Double-clicking on any of the nodes shows the corresponding events.

Blocking may result in a timeout, if a timeout duration is specified. These are displayed as orange labels in Tracealyzer. Timeouts may occur on many Zephyr API calls and are important to keep track of. Some might be intentional, while other may indicate serious errors.

Semaphore Events

Semaphores allow for waking up tasks on events. An example screenshot is shown below. The RX task is initially blocked by a previous semaphore call and wakes up when the semaphore is signaled.

Semaphores are sometimes used to protect critical sections in the code, i.e. mutual exclusion, but mutexes are better for this purpose. When using a semaphore for this purpose there is a risk for priority inversion, meaning that high-priority tasks are delayed by lower-priority tasks. Tracealyzer makes it easy to spot issues like this, for example using the “Actor Instance Graph” showing the task response times.

Mutex Events

Mutex objects are designed for critical sections. Zephyr implements priority inheritance for Mutex objects to avoid priority inversion. The below example shows how Mutexes appear in Tracealyzer. The blue labels show the priority inheritance, where the priority of the holding task is raised (inherited) to the same level as the waiting task to avoid unsuitable preemptions.

This trace view, combined with a large set of visual overviews, makes it easy to understand the real-time behavior of your Zephyr system. You can verify that task priorities are suitable and that the system works as designed. If something seems to be wrong, you can isolate and debug the real-time behavior without halting the system, especially in combination with application logging (see below). You can also profile the system to ensure it runs in an efficient manner, so you get the most out of your hardware.

Tracealyzer provides advanced logging capabilities. Log custom events and data in your application code and display it in Tracealyzer, together with the kernel trace and as graphical plots. This provides deeper observability into the application code at runtime.

Unlike printf-calls, the Tracealyzer logging does not slow down your code by several milliseconds. The efficient logging functions can eliminate over 99% of the logging overhead compared to printf over a UART. This low-impact logging ensures you get the right picture in your debugging, without probe effects from slow logging calls.

For example, state transitions can be logged and displayed in a “logic analyzer” view and as a state diagram. The result can be shown within the trace view (as shown on the left) or summarized as a state graph (on the right), making it easy to spot incorrect behavior.

Learn more how Tracealyzer simplifies debugging with advanced, low-impact logging capabilities.

You do not need any particular hardware to use Tracealyzer, just a suitable development board. You can capture snapshots of the latest activity from a trace buffer in RAM, or stream the data to host using for example Ethernet or a debug probe like SEGGER J-Link, IAR I-jet or Keil ULINK. This way, you can monitor your system over long periods of time and capture any issues.

Tracealyzer for Zephyr relies on the Percepio TraceRecorder, which is available in the Zephyr repository under the Apache 2.0 license. This is integrated in your project within a few minutes by following the Getting Started guides.