Debugging Techniques and Tools

The embedded systems engineer can draw on a variety of tools and techniques for debugging both the software and hardware aspects of the system.

Table of Contents:

  1. The Mental Game
  2. Hardware Debugging
  3. Software Debugging
    1. Understanding How Debuggers Work
    2. Supporting Tooling
    3. Device-side Capabilities
    4. Infrastructure Capabilities
  4. Related

The Mental Game

Much of debugging is a mental game. The following mental strategies are helpful to cultivate.

Hardware Debugging

Embedded programmers, electrical engineers, and system designers need to be comfortable debugging hardware. Requisite skills include:

  • Adding and using test points to gain access to traces for debugging purposes
  • Using equipment to analyze shorts, opens, signal integrity, signal interactions, bus operations, noise, and more:
    • Logic analyzers
    • Oscilloscopes
    • Multimeters
    • Current probes
  • Reworking existing designs to test proposed fixes
  • Use of bus analyzers, such as an Aardvark I2C Adapter, to communicate with ICs and isolate software and hardware problems
  • Creating and using development boards to gain access to additional signals and measurement points, which may not be accessible on the form-factor design

Embedded software engineers need to be familiar with a variety of tools that can be used to debug hardware:

  • JTAG and SWD enable run-time debugging through the use of a hardware adapter
    • OpenOCD is an open-source tool that can be used with SWD and JTAG adapters
    • A variety of commercial solutions also exist, such as J-Link
  • While In-circuit Emulation (ICE) is rarer than it used to be, you can still find ICE devices that provide superior debugging capabilities to a JTAG or SWD approach
  • Protocol adapters can be used to analyze, debug, or send/receive bus traffic
  • FTDI chips, and related clones, can be used to control GPIO externally, communicate with a device using a serial protocol, I2C, or SPI.

Software Debugging

You should be familiar with a debugging tools, such as gdb or llvm.

Understanding How Debuggers Work

Eli Bendersky has a helpful series that explains how software debugging tools work:

Supporting Tooling

The following tools are useful in supporting debugging efforts:

  • Turn up compiler warnings, and make sure that there are no warnings before you start your debugging effort.
  • Sanitizers are valuable debugging tools, but primarily require that you can run your code on a development machine. We use sanitizers with unit test programs and simulators.
  • Static Analysis tools should be incorporated to ensure that your team is not spending time debugging easily identifiable errors.

General approaches that can help with debugging:

Device-side Capabilities

The following device-side capabilities are invaluable when debugging:

  • On-device logging – logging frameworks enable developers to record data at various levels of verbosity at runtime.
    • If logging is not available, you can resort to “printf debugging”, where print statements are used for tracing program execution
  • Reset reason detection – recording the “boot cause”, “reboot cause”, or “boot reason” reported by a processor during the boot process can help identify brown-outs, watchdog resets, or other unexpected reboot causes
  • Assertion support – assert statements can be used to enforce logical conditions at runtime, stopping errors form propagating throughout the system.
  • On-device Crash Dumps – Panics and crashes can be triggered whenever an invalid operation occurs, such as dereferencing NULL or another invalid memory address. Assertion support can also be routed through the crash dump code.
  • Device metrics can be used to observe device state and telemetry information over time.
  • Check-in and Heartbeat Messages can be used to debug intermittent connectivity and to collect metrics

Infrastructure Capabilities

The following infrastructure capabilities significantly aid debugging:

  • A device data collection mechanism is essential for accessing logs, crashes, metrics, and other debug information.

Share Your Thoughts

This site uses Akismet to reduce spam. Learn how your comment data is processed.