Getting Started With Linux Tracing

This page explains how to get started with Linux tracing for Percepio Tracealyzer. Follow the steps below to prepare your target system and your Tracealyzer host. There is also a brief video, but make sure to read the full guide.

1) Installing Tracealyzer on Linux host computers

Tracealyzer requires the Mono framework to run on Linux. See this installation guide.

Also make sure to read RunningOnLinux.txt in the Tracealyzer application directory. This contains additional instructions for certain distributions.

2) Setting up LTTng on the target computer

Install LTTng on your target device. The steps will vary depending on your target system. As a first test, and to familiarize yourself with LTTng, you may try installing it on your host computer. On Ubuntu this is only a few apt-get commands.

Make sure the user running the tracing is part of the tracing group, unless you are tracing as root.

In some cases you need to start lttng-sessiond as root before recording a trace.

When you have installed LTTng and made sure your user is part of the tracing group, you should be able to create a simple trace, for example like this:

lttng create
lttng enable-event -k sched_*
lttng add-context -k -t pid
lttng add-context -k -t ppid
lttng start

# Run some test command to get more activity in the trace
echo “Hello world” | wc

lttng stop
lttng destroy

This should give you a trace in ~/lttng-traces if everything is setup correctly. Now you can load the trace in Tracealyzer by selecting File -> Open -> Open Folder.

3) Viewing LTTng Traces in Tracealyzer

You start Tracealyzer using the launch-tz.sh script found in the Tracealyzer application directory.

If you are new to Tracealyzer, we recommend starting with our article series on Linux tracing at https://percepio.com/tag/tzlinux/.

Next, make sure to read the Tracealyzer User Manual.
This is found in the Help menu. It is recommended to read the sections Views, Tools, Event Mapping, User Events, and Intervals and State Machines. Ignore the section about TraceRecorder, which is intended for RTOS tracing only.

Make sure “Raw Events” are enabled in the settings. See File -> Setting -> Project Settings -> Performance Settings. Make sure that “Include Raw Events in Trace” is checked.

The next step is to acquire an LTTng trace from a target system. To start with, you can open the example trace recorded in the previous step, for example using File -> Open -> Open Folder and select the root of the trace folder.

A more convenient way to record traces is using the Tracealyzer Target Library Tool, found in the Trace menu. This lets you control the tracing and download traces directly from Tracealyzer application, using an SSH connection.

Target Library Tool

Using the Target Library you can setup target profiles for easy access. Use the New button to create a new profile. From there you can setup the address, username and password or private key file. Once a profile has been created, the Connect button will allow you to connect to your target.

A tab will open to the right with three main parts. A trace session control panel, a command log and a command details box. Click the Start New Session button to start a new session. By default, a TzTrace session has been defined which is preconfigured to include scheduling, syscalls, signals and UST events. You can create your own setup using the Sessions And Channels Editor.

Quick Connect options

The Target Library Tool also provide a “Quick Connect” option, found in the lower left corner. This lets you specify all parameters in a string, for example:

host=raspberrypi;user=pi;pass=raspberry;

The complete set of Quick Connect options are

host – Hostname or address of the target
port – SSH port (defaults to 22)
myhost – Address to the host machine as seen from the target. This is can usually be left unspecified, but can be explicitly specified if the target is unable to lookup the hostname of the host computer.
user – The user to connect as
pass – The password of the user
keyFile – Key file to be used with public key authentication. This is highly experimental. Note the uppercase F.
cmdname – The name of the lttng command. Defaults to lttng, but can be overridden in case a prefix is used or similar.

Once a connection string has been entered, you should be able to connect using the button titled “Connect”. This requires your host machine to have the lttng-relayd tool installed, commonly installed from the lttng-tools package of your distribution. If you are running Tracealyzer on a Windows host, you need to use Windows Services For Linux and make sure lttng-relayd is installed and accessible in that environment.

For the best experience, we recommend enabling the pid and ppid context fields on the kernel channel containing the scheduling events. When tracing userspace events (UST), we recommend to also enable the vtid context. This will help Tracealyzer present events correctly under some special circumstances, in particular involving migrations in multicore systems.

Learn more about LTTng and its capabilities at lttng.org.

Special Tracealyzer features for Linux tracing

When viewing an LTTng trace, Tracealyzer offers additional analysis features designed for Linux systems:

The Signals And Syscalls explorer is a tool that can help you analyze how a process interacts with the kernel through syscalls. For example, it will let you track which files are being accessed. It will try to help you track file descriptors (there are some limitations) and it will help you identify failing calls. It can show you a list of syscalls per thread, process or process tree. In addition, it can show you how signals are generated and delivered.

The syscalls plot will let you plot the duration of syscalls. It support two modes, execution time and response time. The former will only account for time when the thread is active while the former will include time when the thread is switched out (i.e. blocked).

The communication flow view has Linux-specific modes that can be accessed through its View menu. It supports a process tree mode, a signals mode and a file descriptors mode. The latter can help you track how processes are communicating over pipes etc.

If you select a raw event in the main trace view or in the event log and right click it, you can add numeric fields to a quick plot. This can be used with any event type, including your own user space tracing events.

Linux Events

Tracealyzer has built-in awareness of the following types of Linux kernel events:

Scheduling events (sched_* in the Kernel domain)
Signal events (signal_* in the Kernel domain)
IRQ and SoftIRQ events (irq_* in the Kernel domain)
Syscalls (--syscall -a in the kernel domain)

Any other events will be presented as “Raw Events”. Such events are not interpreted by Tracealyzer by default, only displayed as Event Labels in the Trace View (i.e. those with cyan background color) and are also listed in the Event Log.

Note that Raw Events can be configured using “Event Interpretations” to create “User Events”. These can be plotted in the User Event Signal Plot and converted into State Machines or Custom Intervals. Read more in the Tracealyzer User Manual under the sections “Event Mapping”, “User Events” and “Intervals and State Machines”.

Known Issues

The Tracealyzer Target Library uses the LTTng Machine Interface (via SSH) to connect to LTTng on the target system. This LTTng interface has limited support for snapshots. However, the Target Library is only provided for convenience. You may also control LTTng manually via a terminal. Then transfer the trace directory to the host computer and use File -> Open Folder.

Traces with uneven buffer coverage can be problematic. This is sometimes the case with snapshot traces on multi-core Linux systems. By uneven buffer coverage, we are referring to scenarios where different LTTng channels or event streams cover different time periods. If you have 30 seconds of scheduling data for one core, but only 5 seconds of scheduling data for another core, these are not necessarily aligned at the start or end of the trace. This makes it difficult for Tracealyzer to synchronize the scheduling trace. This problem is avoided by ensuring that the whole trace is captured, e.g. by streaming the traces to the target file system (store on target).