http://www.percepio.com

Tracealyzer for FreeRTOS

Introduction

Tracealyzer for FreeRTOS is a sophisticated tool for tracing and visualization of FreeRTOS based systems.

Tracealyzer gives an unprecedented insight into the runtime behavior, which speeds up debugging, validation and performance optimization. Issues that otherwise may require many hours, days or even weeks to solve, can with Tracealyzer be understood quickly. This saves you many hours of frustrating trial-and-error troubleshooting. With Tracealyzer you see what is really going on in your system!

"Tracealyzer has always offered exceptional value to our global community of users by giving engineers direct insight into how their applications are executing. This information is invaluable when both optimizing and debugging FreeRTOS applications." - Richard Barry, FreeRTOS founder.

You may have used some tracing tool before, but Tracealyzer is more than a just another trace viewer. Tracealyzer provide state-of-the-art visualization developed since 2004 and is specifically designed for RTOS-based systems. Moreover, Tracealyzer actually understands the meaning of the RTOS events and performs advanced analyses to improve the visualization and simplify comprehension. This includes highlighting of related events, following individual messages between tasks (through message queues) and finding the cause for a particular task activation. This makes it much easier to draw conclusions from the data, understand the problem and verify the solution.

Tracealyzer provides more than 25 interconnected views of the runtime behavior, including task scheduling and timing, interrupts, interactions between tasks, as well as User Events that you may log from your application code. Tracealyzer can be used side-by-side with a traditional debugger and complements the debugger view with a higher level perspective, ideal for understanding real-time issues where a classic debugger is not sufficient.

Tracealyzer does not require additional tracing hardware, which means that it can be used in deployed systems to capture rare errors which otherwise are hard to reproduce. Tracealyzer supports trace streaming via leading debug probes, and since highly configurable also allows for streaming via other interfaces, such as USB, TCP/IP or any other interface you might have available.

System Overview

Tracealyzer for FreeRTOS consists of two main components:

  • The PC application Tracealyzer, that provide trace visualization. It runs on Microsoft Windows and Linux, the latter using the Mono framework.
  • The trace recorder library, for recording of FreeRTOS events.

We also provide integrations with common IDEs and debug probes in order to simply data upload, e.g, Eclipse, SEGGER J-Link, IAR Embedded Workbench, Microchip MPLAB X IDE, and Atmel Studio 7.

Using Tracealyzer

Tracealyzer provides several graphical views which gives different perspectives of the runtime behavior, based on a recording of task scheduling and FreeRTOS kernel service calls. You may also choose to include interrupt service routines (ISRs) as well as application events by adding own calls to the trace recorder library. Application events, or user events, contain a custom text label and optionally also associated data - integers or floating point values.

This section is intended to give a quick overview of the features and how this tool can be used. More detailed information on the various features are available in the later sections.

Also make sure to have a look at the Getting Started page on our website, featuring video tutorials, articles and several examples of how Tracealyzer is used in practice, including real-world cases.

The main trace view provides all recorded information on a vertical time line, focusing on the execution of tasks, interrupt handlers and kernel calls. This view is complemented by some 25 additional views providing high level overviews or focused views from different perspectives.

Tasks and ISRs are commonly referred to as Actors in Tracealyzer (i.e. execution contexts). The execution of actors is shown as solid rectangles in the trace view, called Fragments. A fragment is a continous interval of execution, beginning and ending with a context-switch.

Kernel events such as calls to kernel services and User Events are displayed as text labels on the right side of the fragments. The labels are color coded depending on the type and status of the operation (red for blocking, green for resume, yellow for user events etc.).

Actor Instances are groups of fragments, corresponding to one iteration of the main loop (for tasks) or one execution of an interrupt handler (for ISRs). Instances are shown as a blue outline when selecting an actor in the trace view. The instances contain all events that occurred during this specific execution and include at least one, but often several fragments. The concept of instances allows Tracealyzer to provide detailed timing statistics, for instance in the Actor Instance Graph view.

Tracealyzer creates the instances by recognizing certain events (see below) that are considered to be the finish of the current instance. The next event from that actor is counted as the first event in a new instance.

For ISRs, which are traced explicitly by calling vTraceStoreISRBegin() and vTraceStoreISREnd(), the current instance ends when vTraceStoreISREnd() is called.

For tasks (traced automatically) the "instance finish" is typically finished when calling a kernel function to await the next activation, e.g., using a message queue, semaphore or delay. But there is no "universal truth" for what function calls that should be considered as the "instance finish" for tasks. Therefore, Tracealyzer uses a set of default rules for deciding when instance is considered to be finished, which are "typically correct" but can be overridden or disabled:

  • Task instances finish when explicitly suspended, or waiting on "delay" call.
  • Task instance finish when blocked on a "read" from a message queue, semaphore or similar.
  • Mutex events are not counted as "instance finish", even if blocking.

If not satisfied with the default task instances, you may call vTraceInstanceFinishedNow or vTraceInstanceFinishedNext to explicitly mark the current instance as finished.

In snapshot mode, you may disable the default rules by setting TRC_CFG_USE_IMPLICIT_IFE_RULES to 0 in trcSnapshotConfig.h. This way, no default grouping is done, so unless you call vTraceInstanceFinishedNow/Next for each task, they appear to have just a single long instance.

Additional Views

Below follows a few examples of other views and dialogs which complements the trace view. There are over 25 views in total. All views are connected to the main trace window or another relevant view, which allows you to switch perspectives without losing focus.


To locate a particular task or interrupt in the trace view, open the Finder (Shortcut Key "Ctrl-F"). In the Finder dialog, you can also specify filters on timing properties such as response time, in order to quickly find the extreme cases.


The Object History View allows you to track a particular kernel object, in this case a message queue. You can follow a message from send to receive, or a semaphore signal. You can see the ownership of mutexes. Double clicking in this view highlights the event in the main trace view.

The CPU Load Graph displays the CPU usage per task and interrupt over time. This can be zoomed separately of the main trace view, and can also be used to navigate the trace view by double clicking in the graph.

Example Uses

Make sure to visit the Getting Started page on our website, that provide multiple examples in blog, article and video format.

The general uses of Tracealyzer include:

  • Real-time debugging - Analyze the detailed execution of tasks and interrupts. Why did the system freeze? Why is there sometimes a "glitch" in the behavior?
  • Validation - Is the system executing the way you think it is? Are the tasks and interrupts executing when suitable? Are the task priorities suitable? Is there any undesired blocking or timeouts? How large does the message queues need to be?
  • Profiling - Study the CPU load over time and find hot-spots where tasks are delayed and optimization is relevant. How much processor time and memory is allocated by the different tasks? How is 3rd-party components such as TCP/IP stacks and File Systems behaving? Are there any memory leaks?
  • Application behavior - User Events allows you to log and visualize any event or data from your application code, such as inputs, state transitions or other any variable of interest. User Events are stored in a matter or microseconds, which allows for using them also in time-critical code.

Tracealyzer gives you better possibilities for finding out what is really going on in your system. For instance, if a task never gets to execute you can see why this is the case. Perhaps higher priority tasks are using all CPU time, so that the scheduler never lets the lower priority tasks execute. This is revealed by the CPU Load Graph.

Another reason for a "missing task" might be that the triggering message/semaphore signal is never sent, or sent to the wrong destination, or that the timeout is too short so that the send fails. The Communication Flow graph shows an overview of task interactions. The Finder can be used to find individual events, e.g., a message sent from a particular task to a particular message queue, and has a filter which can be used to quickly find operations which failed due to timeouts.

If the system performance is too slow, or if you simply want to measure the performance, you can study the Statistics Report for a overview of performance metrics like task response times and average CPU usage. Moreover, the Actor Instance Graph reveals the instances with high executions time and/or high response times.

All views are interconnected in clever ways, allowing you to navigate between views with ease and see the data from different perspectives. For instance, if you find a high response time in the Actor Instance Graph, you can simply click on the data point to focus the main Trace View on this point and see the cause - it could be that the actual execution time of the task is unusually long, but could also be caused by interference from other tasks or interrupts. This allows you to focus your optimization efforts on the true cause, which can save you a lot of time.

Requirements

Running Tracealyzer requires a development PC running Microsoft Windows with .NET Framework 4 or later installed, or Linux with Mono installed (included in many distributions). Mono is also available for other platforms, including Mac OS, but this is not guaranteed to work as intended. To record a trace, you also need:

  • A target system running FreeRTOS with the event recorder enabled.
  • Some kind of connection to the target system, typically using your IDE and debugger. It is also possible to stream a trace over USB or other available interfaces by defining you own "stream port".


Copyright Percepio AB 2017, all rights reserved.