Remote Anomaly Detection and Debugging for RTOS Software

Percepio DevAlert® is a cloud-connected monitoring framework for OEMs developing RTOS device software, providing instant insight on bugs and anomalies. Get a diagnostic feedback loop from remote devices to developer and support teams, with visual trace diagnostics that can reduce debugging time by 80% or more. Enable 100x faster reaction time on anomalies in the field and deploy updates rapidly to safeguard customer deployments at scale.

The DevAlert Sandbox makes it easy to experience DevAlert and explore the possibilities it provides. You get a realistic RTOS application directly in your web browser, with DevAlert already integrated. Get started in five minutes, without writing a single line of code.

Read the Solution Brief (PDF) for AWS IoT Core. Other clouds can also be supported.

The core of Percepio DevAlert is a fully managed cloud service, designed to scale to millions of devices without causing alert fatigue or building up massive logs. The solution is designed for security, privacy, scalability, and 100% transparency towards the integrator.

Uploads only occur when anomalies are detected in the device, as decided by the integrator. The data is sent directly to the OEM’s own cloud account, just like other application data and with the same level of security, leveraging the existing network connection. The uploaded traces can be limited to a few kilobytes or even less, and remain under the OEM’s control at all times. The device integration is provided by a C library with full source code.

“DevAlert is a game-changer in that it enables instant feedback from systems deployed in the field, to ensure your firmware quality is constantly improving.”

Jack Ganssle, The Ganssle Group

“The world has been waiting for a solution like DevAlert. This will revolutionize product quality and it’s a must-have for any embedded or IoT project.”

Jacob Beningo, Beningo Embedded Group

Accelerate Device Testing

Imagine that you have 50 devices in your test lab, running tests for many days or weeks. Sometimes you get crashes or weird glitches, but you can’t isolate the problem despite many hours of heroic debugging. The release deadline is approaching and the pressure is on. What do you do?

Having the right diagnostic data is key. Let’s compare with a crime scene, where you are the investigator. If you had a surveillance video of the incident, you could probably solve the case quickly. DevAlert is a “surveillance camera” for your RTOS system. When an anomaly is detected, an alert is uploaded with a software trace for Percepio Tracealyzer. This can reduce debugging time by 80% or more.

DevAlert diagnostics in Tracealyzer

DevAlert is an effective solution for test monitoring, both for manual tests and automated tests in CI/CD pipelines. This enables deep insight on software issues like random crashes and performance issues, as well as cloud-based collaboration within distributed teams across continents and language barriers. Device to cloud connectivity is not required for test monitoring, which makes it applicable for a broad range of RTOS applications.

DevAlert lets you set up alerts on any software condition of interest and provides a trace of the software execution leading up to the alert. This way, you can often see the problem right away. The trace can also tell you how to reproduce the issue, in case you need to dig deeper. This is especially valuable for errors detected during manual testing, where the exact circumstances might be hard to repeat, and for flaky automated tests that occasionally fail without an obvious reason.

The trace includes kernel events like task-switching and kernel API calls, and you may also extend the trace with additional logging in your application code, such as inputs, outputs and important internal variables. Logging events in the trace buffer is often 100x faster than printf logging, since avoiding string formatting and serial output. This way, it is possible to trace the software in detail, several thousand events per second, without noticeable impact.

The device interface for DevAlert is a C library, provided with full source code. It has a footprint measured in kilobytes and can be configured to fit most RTOS systems. This can remain active in deployment, provided some form of connectivity or storage.

To learn more, sign up for a free evaluation and get started with DevAlert Sandbox, directly in your web browser.

Safeguard Deployments at Scale

Say that your company has just launched a new smart device, but more and more customers are reporting that the devices don’t work as intended. A bug is suspected but the developers can’t reproduce the issue for debugging. The customers gets increasingly annoyed and start posting bad reviews, like in the example below. This could seriously harm your customer retention and top-line revenue.

Reviews of a buggy IoT device

Most software has issues remaining at release. According to Software Productivity Research, on average, 5% of the bugs are not detected before release. According to IBM System Science Institute, such anomalies are, on average, 15x harder to solve than if found during development. This because detailed diagnostics isn’t available, unless you have a diagnostic feedback loop from the devices.

Without automated diagnostic feedback, you are “flying blind” and hoping for the best. Relying on your end users to report any issues is ineffective since they don’t always bother to report the issues and cannot provide enough details for debugging. Weeks or months may pass before a solution is hopefully found, while more and more customers are affected every day.

DevAlert provides a diagnostic feedback loop from deployed devices to developer and support teams, enabling OEMs to react 100x faster on software anomalies in the field and gain full insight on the same day. This allows for rapid solutions to safeguard deployed devices at scale.

DevAlert scales to large device fleets, without causing alert fatigue or building up massive centralized logs. Instead of logging and uploading “everything” from every device, DevAlert only uploads when an anomaly has been detected in the device software. The alert includes a detailed trace of the anomaly, showing the most recent events before the anomaly was detected, visualized using Percepio Tracealyzer. This can speed up debugging by 80% or more.

The alerts are summarized by the DevAlert Dashboard, providing a compact “to-do” list of the underlying issues in the code. Even with millions of devices in the field and thousands of alerts per day, the Dashboard will still be clear and manageable. It makes new issues stand out and ensures nothing is missed. And you can see how many times each anomaly has occurred, which helps prioritization if you have multiple issues at the same time.

Imagine how much frustration and debugging time that can be saved by automatic, instant notifications directly to the developer team, on the very first occurrence of the issue, and with detailed diagnostics that shows what happened.

DevAlert is also valuable for devices without cloud connectivity. It is an effective solution for test monitoring, and can also be used in deployed devices without direct cloud connectivity. Alerts can be stored locally and uploaded later, e.g., when connecting a laptop over USB or a smartphone over Bluetooth.

To learn more, sign up for a free evaluation and get started with DevAlert Sandbox, directly in your web browser.

DevAlert in Action – Demo on Cypress PSoC64