5 Steps to Speed Multithreaded Embedded Software Development

Feb 15, 2021 |

Your organization can get to market faster with higher-quality products when given better insight into the “dark side of the code”—the actual behavior of the full software system.

Intended and actual behavior may differ in myriad ways that are not apparent from the source code and are difficult to detect by functional testing. This is an increasing challenge for developers of multithreaded embedded software, running an RTOS or Linux kernel, when the applications become increasingly complex.

Follow these five best practices to gain the visibility you need into real-time behavior at the system level to improve product quality and speed up development.

1.    Reduce guesswork with a visual timeline.

Problems detected when running a full multithreaded system are often difficult to resolve. Fortunately, better insight during debugging results in less guesswork. This makes it easier to find the root causes, so the bugs are be fixed properly and the debugging takes less time. The ability to see a visual timeline of software execution is important in many scenarios, such as when the location of the bug is not obvious from the initial symptoms, when the issue is difficult to reproduce, and when traditional debugging tools (halting the execution) can’t be used.

What about printf debugging? The old practice of adding debug prints in the application code is risky, intrusive and does not scale to more complex applications and faster processors. The printf function is not designed for this purpose and is typically quite slow—several milliseconds per message. In contrast, visual trace diagnostics relies on optimized solutions for software tracing and is about 100x faster than printf calls, so you can collect a lot more information without noticeable impact. Moreover, data visualization techniques make it easy to turn software traces into actionable insights.

2.    Monitor system tests to catch elusive issues.

Continuous live streaming of trace data enables developers to record and visualize a complete picture of the software over long periods of time to capture elusive software issues that otherwise are difficult to reproduce and debug. Capture any “glitches” in a trace recording and learn what is causing them. Or find out why the system crashed after hours of stress testing. The visual trace diagnostics of Percepio® Tracealyzer® enable you to see live animated views of the runtime system, spot anomalies in high-level visual overviews and statistics, and drill down to the specific events to see exactly what happened.

3.    Improve software design for better products and lower costs.

Some development projects run into “debugging hell,” a common scenario in which massive amounts of debugging impede the project. Deviations from best practices in software design and RTOS application development are often major contributing factors. Some examples include unsuitable task priorities, large variations in software timing and too many dependencies between the tasks. Such issues may cause complex and chaotic system behavior with poor testability, which increases the risk for elusive bugs that leak out into production devices.

Visual trace diagnostics helps the developer analyze and improve the software design and ensure stable and reliable system behavior. When you can find software design flaws earlier, fewer changes are required to fix them and the risk of bugs in shipped products is reduced. Software design improvements can also result in dramatically improved system performance and responsiveness. This can allow for selecting more cost-effective processors to reduce BoM costs, or use a lower clock frequency for longer battery life.

4.    Verify timing and product performance throughout the project.

Measure timing and performance metrics as early as possible, so you can address any issues before delivery. This is crucial for real-time systems with explicit timing requirements and important for the user experience of other applications. No one likes a sluggish touch screen interface or a slow Wi-Fi router, but often the causes are not obvious in the source code, and a faster processor might not help much if the problem is rooted in software design.

Visual trace diagnostics makes it easy to show the cause of anomalies and to measure software timing properties, memory usage and CPU load. Easier compliance with real-time requirements and better application performance and customer satisfaction are some of the many benefits.

5.    Make diagnostics easily available and benefit every day.

Visual trace diagnostics can be provided without need for additional hardware, using only a software integration and the existing hardware of the application. This way, you can include it by default in software systems, so developers have access throughout development, testing and release. When you make diagnostics readily available anytime an issue is encountered, all developers can benefit every day. Log any information of relevance to your application, including internal data and states that aren’t exposed during runtime. See any data plots desired, in parallel with the visual execution timeline, and make a great product that beats the competition.

Want to learn more about visual trace diagnostics?

Percepio® Tracealyzer® is the premier solution for visual trace diagnostics, providing a visual timeline of embedded software at runtime for developers of RTOS- and Linux-based systems.

Percepio® Tracealyzer® is an easy “plugin” to your embedded software development that increases your team’s ability to deliver high quality products on time and within budget.

Tracealyzer works with essentially any processor and compiler—with no extra hardware required.

Test drive Tracealyzer for 10 days

Do you have questions? Let’s discuss your challenges and goals and how Percepio® can help. Contact us.


Percepio is the developer of a highly visual runtime diagnostics tool for embedded and Linux based software, Tracealyzer, and of the award-winning DevAlert, a cloud-based service for error reporting in deployed IoT devices with Tracealyzer diagnostics. Percepio collaborates with several leading vendors of operating systems for embedded software and is a member of the Amazon Web Services Partner Network (Advanced Technology Partner).

For systematic use of visual trace diagnostics, see also Percepio DevAlert. Get automatic alerts directly from devices to developers during testing and in deployment, with visual trace diagnostics to speed up resolution.

Percepio® DevAlert® and Percepio® Tracealyzer® are registered trademarks.