This text assumes that you have DevAlert integrated in your AWS account. To get started, apply for a DevAlert evaluation and follow the provided instructions.

DevAlert Firmware Monitor (DFM)

The device side of DevAlert is called DevAlert Firmware Monitor (DFM). This is provided in C source code, that needs to be integrated in your device source code. DFM provides an API for generating alerts and has one major subcomponent, the TraceRecorder library, which continuously records the software execution into a ring-buffer in RAM. This trace data is included with the Alerts to allow for visual trace diagnostics with Percepio Tracealyzer.

The DFM library is used to generate Alerts from your device source code, e.g. when errors are detected. This is typically done by calling the DFM library from existing error handling code. However, DevAlert is not just for errors, but can be used for any kind interesting condition detected in your code. For instance, you may use Alerts for proactive warnings, such as “Heap memory usage exceeding 95%”. And if your software is measuring key performance metrics, you can use Alerts to report situations with low performance and thereby see the cause in the included trace. For instance, if the startup time exceeds a certain threshold.

Note that Alerts are intended for occasional issues, not for frequent events, since secure data upload takes significant time on small IoT devices (tens of milliseconds). If you are interested in fast data, e.g. a sensor reading that occurs every 10 ms, you can include the values in the trace as “user events”, in a matter of microseconds. Such user events can then be plotted and analyzed in Tracealyzer. In this way, you only need to generate a single Alert when an interesting condition is detected, and you get the full sequence of the most recent data points.

When an Alert is to be generated, you may either upload it to the cloud directly or store it to non-volatile memory (e.g. a file system, or directly to internal flash memory). The latter is valuable in case of severe errors, where the device needs to be restarted before an upload is possible. Such stored alerts will be uploaded automatically after restart, when the initialization function is called.

DFM file structure:

  • percepio_dfm.c/h: public API for initializing DFM and generating Alerts.
  • percepio_dfmConfig.h: configuration options for the DFM Library.
  • dfmKernelPort.c/h: interface towards the kernel; can be modified to fit another RTOS kernel.
  • dfmCloudPort.c/h: defines how to upload data to the cloud.
  • dfmCloudPortConfig.h: configuration options for the cloud port.
  • dfmHardwarePort.c/h: defines how to store data in non-volatile memory; can be replaced to work with other MCUs or storage methods.
  • dfmCodes.h: defines the Alert codes and Symptom codes; should normally be generated from the DevAlert cloud service.

The core concepts of DevAlert that you need to understand as a developer are:

  • Alert: an individual report uploaded by the DFM library. Contains the alert signature and a software trace.
  • Alert signature: data provided to the central DevAlert cloud service. This includes
    • Alert type: a numerical code describing the type, e.g. “Out of memory”, “Stack overflow” etc.
    • Symptoms: additional details about an alert, e.g. the location in the program code. Symptoms are numerical key/value pairs, and the values are 32-bit integers.
    • Software revision: version of the firmware running on the device. Needed to allow filtering of alerts per revision.
    • Device name: a unique ID for the device. Needed to to allow filtering of alerts per device.
  • Issue: all alerts with the same alert type and symptoms are grouped into an Issue.

You can define your own alert types and symptoms to match your needs, but you should not edit dfmCodes.h directly. The DevAlert cloud service needs to be aware of your definitions to show the Dashboard correctly, so instead we provide features in Tracealyzer where you can add new alert types and symptoms. From there you can also generate a new dfmCodes.h file. In Tracealyzer, select Cloud -> AWS -> AWS Description Manager to add new alert types and symptoms. Then select Cloud -> AWS -> Generate dfmCodes.h and overwrite your previous header file. You will now have numerical IDs for your new alert types and symptoms that should be used when calling the DFM library.

Follow these steps to integrate the DFM library in your project:

  1. Make sure your project can record trace data in snapshot mode. See the TraceRecorder section, below.
  2. Add the DFM library source code to the project and put the header files where they can be found by the compiler.
  3. Review percepio_dfmConfig.h and dfmCloudPortConfig.h and modify if necessary to fit your system.
  4. Add a call to ulDfmInit() after a connection to the cloud has been established.
  5. Create an alert using ulDfmAlert() and ulDfmAddSymptom().
  6. Finally call ulDfmSendDataToCloud() or ulDfmSaveDataToNonVolatileMemory() to finalize the alert. The latter function is used on severe errors, where the device needs to be restarted before an upload is possible. Such stored alerts will be uploaded automatically after restart, when ulDfmInit() is called. Note that this function depends on dfmHardwarePort.c, so this file must be updated to reflect your system.

See the Example section below for more information, and the API documentation in percepio_dfm.h.

TraceRecorder library

The DevAlert Firmware Monitor includes the Percepio TraceRecorder library in order to provide software traces with the alerts. The trace data may include both RTOS kernel events (e.g. scheduling events and API calls) and custom “user events” logged by your application code.
The trace data is in a compact, binary data format intended for Percepio Tracealyzer which provides visual trace diagnostics. This is very useful for debugging and for general analysis of runtime behavior. The traces provide important context about the Alert – what was going on in the software when the problem was detected? The trace may reveal the problem directly, or at least help you replicate the problem in the lab. Note that you may add user events in your application code to log additional information, for instance input values and important state changes.

The TraceRecorder library has been refined since 2009 and is trusted by hundreds of development teams world wide. It designed to be highly memory efficient.

For DevAlert, the recorder should be configured for snapshot mode, where the trace data is kept in a RAM ring buffer. To integrate the recorder library, follow the getting started guide for FreeRTOS.

You can verify that the snapshot trace recording works as intended by using our Eclipse plugin. It is compatible with most Eclipse-based IDEs.

Tracealyzer

Tracealyzer is the desktop application that is used to visualize and analyze trace data collected from the device. For more information about Tracealyzer see here. From version 4.4.0 Tracealyzer includes DevAlert support, most notably a DevAlert Dashboard on the welcome page. Note, however, that you need the special license key that is included in the DevAlert evaluation package to enable DevAlert integration.

When you have received the DevAlert evaluation package, install and start Percepio Tracealyzer. Enter your license key and start the DevAlert Connection Wizard, which is available on the welcome screen. This requires two configuration files provided with the evaluation package.

Once the DevAlert Connection Wizard has finished, the DevAlert dashboard should load and show a table with your recent alerts. The table is updated in real time, so now is the time to start executing your application on the board. Any DevAlert reporting you have added to your code should appear in the table in about 10 seconds. The table has one row per issue, and the row color indicates the time since the last alert.

As of September 2020, DevAlert has been tested with FreeRTOS and ThreadX. It should also work with SafeRTOS, Micrium µC/OS-III and VxWorks with minor adjustments, but this has not yet been verified. These and other forms of RTOS support can be provided on request.

Integration of TraceRecorder library

Follow these steps to integrate DevAlert into an existing project. This assumes you are using FreeRTOS and AWS IoT Core.

  1. Integrate the TraceRecorder as described in this quick start guide. More information can be found in the Tracealyzer user manual.
  2. Verify that the TraceRecorder works as intended in snapshot mode, e.g. by using Percepio’s Eclipse plugin.
  3. Add the DFM library source code to the project and put the header files where they can be found by the compiler.
  4. Review percepio_dfmConfig.h and dfmCloudPortConfig.h and modify if necessary to fit your system.
  5. Call ulDfmInit(…) right after the point where the application has connected to the cloud.
  6. Add calls to generate Alerts at suitable locations, e.g. in existing error handlers. See below for an example. Details can be found in percepio_dfm.h.
  7. Add #include “percepio_dfm.h” in all .c files that use the DFM library.

Example Code

Below is an example showing how to generate an alert in case of a failed malloc call (out of heap memory).

/* Error handler - Failed to allocate requested memory */
void vApplicationMallocFailedHook()
{
	// Adding a user event to the trace buffer
	vTracePrintF(devalert_user_event_channel, "Malloc failed.\r\n");

	// Pause tracing during upload
	vTraceStop();

	// Compose the Alert
	ulDfmAlert( DFM_ALERT_MALLOC_FAILED );
	ulDfmAddSymptom( DFM_SYMPTOM_PRODUCT, APP_PRODUCT_IDENTIFIER);
	ulDfmAddSymptom( DFM_SYMPTOM_CURRENT_TASK, (uint32_t) pcTaskGetName(NULL));
	ulDfmAddSymptom( DFM_SYMPTOM_STACKPTR, (uint32_t) __get_PSP());

	// Upload the Alert
	ulDfmSendDataToCloud();

	// Resume tracing
	vTraceClear();
	uiTraceStart();

	/* Ordinary error handling follows, restart? */
}

This example makes use of several important DFM functions:

  • vTracePrintF adds a user event to the trace, in this case the string “Malloc failed”. To learn more about user events, search for user event in the Tracealyzer manual.
  • vTraceStop stops the trace recorder so that no new events are recorded while we generate the alert. The trace buffer should not be modified while reading from it.
  • ulDfmAlert starts a new Alert and sets the alert type. The alert types are defined in dfmCodes.h. Note that dfmCodes.h needs to match the definitions in the DevAlert cloud service, so it is recommended to generate this header file from the cloud service, using the provided code generation feature.
  • ulDfmAddSymptom adds symptoms to an alert. Symptoms provide additional information about the alert, for example the location where the error occurred. Symptoms only allow for numerical data (32-bit integers or addresses). Each symptom has a numerical ID, defined in dfmCodes.h, and multiple symptoms can be added to an alert. Like alert types (see above), symptoms should normally be defined through the cloud service.
  • ulDfmSendDataToCloud sends the alert to the cloud. If the error cannot be recovered and the system needs to be restarted before an upload is possible, use ulDfmSaveDataToNonVolatileMemory instead. This function saves the data to non-volatile memory; the data is then uploaded to the cloud after the system has restarted and reconnected to the cloud.
  • vTraceClear clears any old trace data.
  • uiTraceStart starts the trace recorder again.