The real test of IoT devices comes after shipping, when thousands of end users start using your product, sometimes in unexpected and untested ways. No software is entirely bug free, so some users will run into those remaining bugs. The question is how many and what you do about it.
Follow these five best practices to reduce development risk, find out about bugs before most of your customers do, and get the detailed information your developers need for efficient, continuous software improvement across a fleet of deployed IoT devices.
1. Expect shipped products to have bugs and prepare to deal with them.
Even if your product has been tested thoroughly and no more issues are found, chances are many more bugs remain. According to academic research, between 50-100 bugs are introduced per thousand lines of shipped code and about 5% remain in deployment. While some bugs are harmless, about 20% are major defects. Assuming your project has 50,000 lines of code, then 25-50 major defects likely remain in the product shipped to customers.
These bugs are often elusive issues that only show up under certain circumstances. Otherwise, the bugs would not have been missed by the testing. Customer support might receive complaints and returns, but it is not always clear whether the cause is a software issue, a hardware issue or user error. Moreover, users are often unable to articulate useful information about the problem they experienced, so your developers may receive minimal information with which to reproduce a problem.
Make sure your embedded software provides appropriate diagnostic information in case of any runtime errors or warnings. This can be achieved by logging diagnostic messages about the software behavior to a circular buffer in RAM. When an issue is encountered in the code, the most recent diagnostic data can be copied to non-volatile memory, including specific information about the issue and the events that led to the problem. This information makes it much easier for developers to understand the root cause of the problem and solve it properly.
2. Provide automatic alerts to the developer team.
Storing diagnostics on the device is a great step forward, but you are still dependent on the tester or customer to provide the information. By receiving automatic alerts and diagnostics from the devices themselves with the Percepio® DevAlert® device monitoring solution, developers can begin analyzing the problem within seconds of the very first occurrence.
During system testing, developers can be notified immediately about issues and get detailed diagnostics, without relying on the tester to provide the exact circumstances. This can speed up resolution tremendously, especially in cases where the problem isn’t obvious from the symptoms.
This approach also works for system testing of general embedded software without cloud connectivity. The alerts and diagnostic data can be uploaded to a local computer that forwards the data to a central monitoring service.
Secure IoT connectivity enables diagnostic alerts from the field. This provides real-world feedback from IoT devices and enables rapid, continuous improvement of the device software.
Your development team will then know about real-world issues within seconds after the very first occurrence, instead of being unaware for weeks or months before some affected user reports it. When you learn about issues in the field and fix them quickly, fewer customers will experience the problem. Improving the quality and reliability of the IoT device leads to a variety of good outcomes: higher customer satisfaction, better product reviews, lower product support costs and fewer product returns.
3. Monitor key performance metrics.
Feedback on real-world IoT device performance enables improvement of key performance metrics and the overall user experience, so you can make a great product that beats the competition.
Most developers want to know about runtime errors and warnings, which could arise because of software bugs or even hardware glitches that result in unexpected return values from the driver code. Maybe you also want a proactive warning when the stack usage of a task is close to the limit.
But there are much broader applications for automatic alerts. A developer can define custom alert types that capture any condition your software can detect. Maybe you want to measure product performance, such as startup time and battery life, and provide an alert if a certain threshold is exceeded? You could even find out what features are used by customers and make well-informed decisions on future improvements of the product.
Percepio® DevAlert® compares every alert with earlier alerts from your entire device fleet and groups them into specific issues, shown on a dashboard. This provides a clear overview and avoids repeated notifications about the same issue. Any repetitions of previously detected issues are counted and shown in the dashboard, which allows for issue prioritization.
4. Use visual trace diagnostics to speed up resolution.
The diagnostic data may contain a detailed software trace, showing the exact sequence of software threads, API calls, state changes and interrupts. Analysis of such information is greatly facilitated by visual trace diagnostics. This provides both a visual timeline of the events and multiple visual overviews from different perspectives, so any anomalies stand out.
Alerts from Percepio DevAlert include software traces directly accessible from the dashboard. These are easy to analyze with Percepio Tracealyzer®, an advanced tool for visual trace diagnostics integrated with DevAlert that provides more than 30 interactive and interconnected views.
5. Deploy over-the-air (OTA) updates quickly before many customers are affected.
Software issues in deployment typically don’t affect all users at once, otherwise testing would have found them. With IoT device monitoring, you become aware of any remaining issue on the very first occurrence, before most of customers notice. The sooner you can deploy an update, the fewer customers will be affected. IoT device monitoring lets you take full advantage of cloud-based over-the-air (OTA) updates to achieve rapid continuous improvement of your deployed IoT devices.
Want to learn more about IoT device monitoring for continuous software improvement?
Secure IoT cloud connectivity enables IoT device monitoring at scale. Percepio DevAlert offers IoT device developers a new way to achieve continuous software improvement and take full advantage of over-the-air updates for reduced deployment risk.
Percepio® DevAlert® provides IoT device monitoring for use in system testing, field testing and deployment. DevAlert notifies developers about abnormal conditions detected in the device software, with visual trace diagnostics to speed up resolution.
Get a complimentary consultation with DevAlert
Do you have questions? Let’s discuss your challenges and goals and how Percepio can help. Contact us.
Percepio® DevAlert® is a cloud-based monitoring service that makes it easy for device software developers to report any unexpected condition, like errors and warnings. Used in both system testing and in customer deployment, DevAlert returns automatic real-world feedback to the developers within seconds after the very first occurrence.
Percepio® DevAlert® are registered trademarks.