Let’s face it: we can never be certain that any software is free of bugs. On average, 5% of the original bugs remain in the production code, despite all verification efforts. This can mean hundreds of bugs in the shipped product, some are perhaps insignificant or very unlikely, but you can’t know for sure. Testing can’t cover all possible paths through the code; there are simply far too many – especially when considering context-switching between RTOS tasks and interrupt handlers. Code reviews and static analysis help, but these are complementary methods which do not take real-time behaviour into account.
The real test comes when thousands of people start using your product, in ways you never anticipated or tested. Will it stand the test of reality? If not, the remaining bugs will at the very least irritate your customers, damage the reputation of your brand and hurt your sales. In some cases, bugs may even lead to accidents, product recalls and legal action.
So, what can be done? You can always spend more time and money on verification, but the effort needed to find the last remaining bugs increases exponentially. Another possibility comes with the increasing use of Internet-of-Things (IoT) operating systems – Over-the-Air (OTA) software updates. If you fix the problem quickly via an OTA update, your customers will be more satisfied and many of them might never even notice the issue. The faster you can fix a problem through an OTA software update, the fewer customers will notice it.
A ground-breaking new cloud service
However, you cannot fix bugs that you are not aware of. Automatic feedback is needed to make developers aware of firmware issues in deployed products.
Therefore, Percepio has announced Percepio Device Firmware Monitor (DFM), a ground-breaking new cloud service for IoT product organizations that provides awareness of missed bugs and speeds up resolution. DFM notifies the firmware developers within seconds after an error has been detected, providing the error message and a software trace for Percepio Tracealyzer. The trace shows you what was going on in your code when the error occurred and allows for timely resolution. DFM can be used both during testing and in shipped products.
Without an automated feedback system like DFM, you actually rely on your end users to report any remaining bugs. This way, you might not hear about issues until it’s too late, when many customers have already been affected. Moreover, your end users typically can’t provide sufficiently detailed information for you to quickly identify and solve the problem. A vague error report like “the screen went black” may require weeks of guesswork until you find a likely cause, and even then, you still don’t know if you solved the right problem.
Initially, Percepio Device Firmware Monitor supports Amazon Web Services (AWS IoT Core) and Amazon FreeRTOS, but support for additional platforms is planned and can be provided on request. The customer firmware needs to include Percepio’s trace recorder library in order to provide software traces. This uses a very memory efficient encoding and stores decently long traces (hundreds of milliseconds) using only a few kilobytes.
Traces and alerts
DFM relies on your existing secure cloud connectivity and storage. Only an anonymized signature of the issue needs to be provided to the DFM cloud service, which runs in Percepio’s AWS account. This signature does not include the software trace, which is always kept within the customer’s storage. The information provided to the DFM service is completely transparent to the customer. The integrated DFM support in Tracealyzer allows for accessing the reported traces directly from Amazon S3, making it very convenient to use.
Trace data is only uploaded when an error is detected, so this does normally not produce any extra data traffic. Due to the small amount of data required and today’s cheap cloud computing services, the added operational costs of DFM are negligible. Even in a really pessimistic scenario, let’s say you have a million devices out there producing 5 000 error reports per day at 5 Kbytes each, this only results in about 9 GB per year. Storing this in Amazon S3 costs less than $3 per year, even if you never delete any reports. Sending out OTA updates costs a bit more, but this might not be required every time. It is up to the customer to decide if an OTA update is motivated. In case of less critical issues, you may package several bug fixes into a single OTA update.
Early adopters welcome
Does this sound interesting? Please drop us a line at info@percepio.com and we’ll talk! We are looking for early adopters to give us feedback on the DFM cloud service before it launches to the public later in 2019.
You are also welcome to Percepio’s booth at Embedded World, number 4–305, where we will show DFM live for the first time.