The embedded systems engineer can draw on a variety of tools and techniques for debugging both the software and hardware aspects of the system.
Table of Contents:
The Mental Game
Much of debugging is a mental game. The following mental strategies are helpful to cultivate.
- When Stuck, Get a Fresh Perspective
- Keep a Debug Audit Trail to make sure you aren’t going in circles
- Perform a Binary Search to Narrow the Problem Space
Hardware Debugging
Embedded programmers, electrical engineers, and system designers need to be comfortable debugging hardware. Requisite skills include:
- Adding and using test points to gain access to traces for debugging purposes
- Using equipment to analyze shorts, opens, signal integrity, signal interactions, bus operations, noise, and more:
- Logic analyzers
- Oscilloscopes
- Multimeters
- Current probes
- Reworking existing designs to test proposed fixes
- Use of bus analyzers, such as an Aardvark I2C Adapter, to communicate with ICs and isolate software and hardware problems
- Creating and using development boards to gain access to additional signals and measurement points, which may not be accessible on the form-factor design
Embedded software engineers need to be familiar with a variety of tools that can be used to debug hardware:
- JTAG and SWD enable run-time debugging through the use of a hardware adapter
- While In-circuit Emulation (ICE) is rarer than it used to be, you can still find ICE devices that provide superior debugging capabilities to a JTAG or SWD approach
- Protocol adapters can be used to analyze, debug, or send/receive bus traffic
- FTDI chips, and related clones, can be used to control GPIO externally, communicate with a device using a serial protocol, I2C, or SPI.
Software Debugging
You should be familiar with a debugging tools, such as gdb or llvm.
Understanding How Debuggers Work
Eli Bendersky has a helpful series that explains how software debugging tools work:
- How debuggers work: Part 1 – Basics
- How debuggers work: Part 2 – Breakpoints
- How debuggers work: Part 3 – Debugging information
Supporting Tooling
The following tools are useful in supporting debugging efforts:
- Turn up compiler warnings, and make sure that there are no warnings before you start your debugging effort.
- Sanitizers are valuable debugging tools, but primarily require that you can run your code on a development machine. We use sanitizers with unit test programs and simulators.
- Static Analysis tools should be incorporated to ensure that your team is not spending time debugging easily identifiable errors.
General approaches that can help with debugging:
- Continuous observability for debugging RTOS-based firmware by Johan Kraft
Device-side Capabilities
The following device-side capabilities are invaluable when debugging:
- On-device logging – logging frameworks enable developers to record data at various levels of verbosity at runtime.
- If logging is not available, you can resort to “printf debugging”, where print statements are used for tracing program execution
- Reset reason detection – recording the “boot cause”, “reboot cause”, or “boot reason” reported by a processor during the boot process can help identify brown-outs, watchdog resets, or other unexpected reboot causes
- Assertion support –
assertstatements can be used to enforce logical conditions at runtime, stopping errors form propagating throughout the system. - On-device Crash Dumps – Panics and crashes can be triggered whenever an invalid operation occurs, such as dereferencing
NULLor another invalid memory address. Assertion support can also be routed through the crash dump code. - Device metrics can be used to observe device state and telemetry information over time.
- Check-in and Heartbeat Messages can be used to debug intermittent connectivity and to collect metrics
Infrastructure Capabilities
The following infrastructure capabilities significantly aid debugging:
- A device data collection mechanism is essential for accessing logs, crashes, metrics, and other debug information.
Related
- Debugging Process describes a general debugging process
- Debugging Case Studies can provide insight into how others approached particular problems
