Embedded Artistry was founded with the goal of creating reliable, safe, and well-tested embedded systems. The sad fact is that most embedded software that we’ve encountered is low-quality and untested. Perhaps this holds true for most of the software industry – the continual procession of hacks, flaws, and errors is discouraging.
We are focused on ways that we can all improve at our craft. Testing our systems, especially in an automated way, is a direct approach at identifying bugs early and improving our software quality.
We decided to make our list of testing resources publicly available. If you have any favorites which aren’t listed here, write to us!
Table of Contents:
- Unit testing
- Debugger-based testing using metal.test
- Phil Koopman’s lectures on embedded software quality and testing
- Recommended Reading
Unit Testing
Most developers know they should be writing unit tests as they develop new features. If you’re unfamiliar with the concept, unit testing focuses on testing the finest granularity of our software, the functions and modules, in an independent and automated manner. We want to ensure that the small pieces operate correctly before we combine them into larger cooperating modules. By testing at the finest granularity, we reduce the total number of test combinations needed to cover all possible logic states. By testing in an automated manner, we can ensure that any changes we make don’t introduce unintended errors.
“A key advantage of well tested code is the ability to perform random acts of kindness to it. Tending to your code like a garden. Small improvements add up and compound. Without tests, it’s hard to be confident in even seemingly inconsequential changes.”
-Antonio Cangiano (@acangiano)
Unfortunately, at Embedded Artistry we’ve only worked on a handful of projects that perform unit testing.
The primary reason for this is that many developers simply don’t know where to start when it comes to writing unit tests for embedded systems. The task feels so daunting and the schedule pressures are so strong that they tend to avoid unit testing all together.
James Grenning and TDD
James Grenning has put a tremendous amount of effort into teaching embedded systems developers how to adopt TDD. He published an embedded systems classic, Test-Driven Development for Embedded C and conducts TDD training seminars.
James has written extensively about TDD on his blog. Here are some of our favorite posts:
- TDD Guided by ZOMBIES – walks through testing a circular buffer
- Physics of Test Driven Development
- Manual Test is Unsustainable
- I’ve got integration and system tests, why do I need unit tests?
- Preventing Brittle Tests
You can also watch these talks for an introduction into the how and why of TDD:
- Webinar – Test-Driven Development for Embedded Software
- NDC conference talk: Test Driven Development in C/C++
I took James Grenning’s Remote TDD training webinar and wrote about my experience on the blog.
If you’re interested in taking a training class, you can learn more on James’s website.
Matt Chernosky and Electron Vector
Matt Chernosky, who runs the Electron Vector blog, is an invaluable resource for embedded unit testing and TDD. If you are having a hard time getting started with unit testing and TDD, Matt’s articles provide a straightforward and accessible approach.
Here are some of our favorite articles from Matt’s blog:
- 7 tips for adding unit tests to existing firmware
- Mocking hardware interfaces using Ceedling and CMock
- Practice writing code without the hardware
- When you want to unit test, create well-defined software units
- When you want to unit test, abstract the hardware
- Event-based interfaces for testability
- Avoiding mocks by enqueueing events
- Test-driving with mocks instead of hardware
Matt published a free guide for using Ceedling for TDD. If you have experience with TDD but are looking to improve your Ceedling skills, check out his Ceedling Field Manual.
Throw the Switch
Throw the Switch created the Ceedling, Unity, and Cmock unit testing suite. This is the trio of tools that Matt Chernosky uses and writes about.
Aside from creating testing tools, Throw the Switch maintains a library of test-related articles and a course called Unit Testing & Other Embedded Software Catalysts. Additional courses related to unit testing for embedded systems are being developed.
Quantum Leaps
Miro Samek, creator of the QP C and C++ frameworks, has also produced a unit testing framework called QUTest. QUTest is designed specifically for embedded systems and works with C and C++. The framework also provides support for dual-targeting tests to that they are able to run both on the host machine and on the target hardware.
Unlike many of the other test harnesses we discuss, QUTest brakes with the xUnit framework tradition. QUTest works similarly to “debugging with printf
“, using output from printf
-like statements to evaluate the results. The target program runs a test fixture that exercises the code-under-test and produces QPSpy output. Checking the test assertions against expectations is handled on the host machine using test scripts. QUTest provides other interesting capabilities, such as testing Design by Contract assertions on the device.
The QUTest documentation is comprehensive, and there are multiple tutorials to help you get started with the framework.
Unit Testing Frameworks
Listed below are frequently recommended unit testing frameworks for C and C++. There are more unit testing frameworks in existence than we can ever review, so again this list is not exhaustive. If nothing jumps out at you in this list, keep looking to find one that fits your team’s development style.
We already mentioned the Throw the Switch C unit testing frameworks: Unity, Cmock, and Ceedling.
Cmocka is the C unit testing framework we started with. Cmocka ships with built-in mock object support and operates similarly to Unity & CMock. The framework is built with C standard library functions and works well for embedded systems testing.
Catch appears to be the most popular C++ unit testing framework. Catch is a header-only library which supports C++11, C++14, and C++17.
Doctest is similar to Catch and is also header-only. Doctest differs from Catch in that it enables you to write tests in headers.
GoogleTest is Google’s C++ unit testing framework. GoogleTest is one of the few C++ frameworks with built-in mocking support.
CppUTest is a C++ test suite that was designed with embedded developers in mind. This framework is featured in James Grenning’s book Test-Driven Development for Embedded C. C++ features within the framework are kept to a minimum enabling it to be used for both C and C++ unit testing. CppUTest has built-in support for mocking.
If you’re interested in mock object support for C++, check out GoogleMock, Trompeloeil, and FakeIt. Each of these mocking frameworks can be integrated with the unit test frameworks mentioned above.
Other Resources
If you’re brand-new to TDD, read through this walkthrough to get a sense of the approach: Test Driven Development: Example Walkthrough.
Steve Branam, who writes at Flink and Blink, has written posts on testing:
- Testing is how you avoid looking stupid
- Off-Target Testing and TDD for Embedded Systems
- Review: Test Driven Development for Embedded C, James W. Grenning
Steve recommended Jeff Langr’s
Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better as another resource for embedded C++ developers.
Embedded Debugger-Based Testing
We always want to run as many tests as possible on a host PC when unit testing embedded systems code. However, we can’t test every aspect of our system on a host machine. Before shipping the final system, we need to evaluate the target compiler, issues which only present themselves on the target (e.g. endianness, timing), and the actual target hardware and hardware interfaces.
Metal.test is a framework which can help us with on-target testing. Klemens Morgenstern, an independent contractor and consultant, maintains this project. Metal.test enables automated execution of remote code using debugger hardware, such as J-LINK or ST-Link. The project supports gdb
, and he has lldb
support planned for a future release.
Metal.test features:
- I/O Forwarding
- Code Coverage
- Unit testing
- Call tracing
- Profiling
- Function Stubbing at link-time
- Extensive documentation (in the GitHub repository Wiki)
Metal.test also includes a plugin system. While it’s not essential, plugin support enables developers to extend the functionality to support any use case that a debugger can support.
Klemens is looking for feedback on metal.test. Don’t hesitate to reach out with questions, issues, or other feedback.
Phil Koopman on Testing and Software Quality
Phil Koopman is a professor at Carnegie Mellon University. He also runs the Better Embedded Software blog, which is a must-read for embedded developers.
Phil has produced an immense and invaluable body of work, much of it focused on embedded software quality. The lecture notes for his embedded systems courses are available online, and he posts lecture videos on his YouTube channel.
Here’s a selection of his lectures related to testing and software quality:
- Embedded Software Quality: Why is it so terrible? What can we do about it?
- Embedded Software Testing Overview
- Unit Testing
- Integration Testing
- Testing Quality
- System Level Testing
- SQA isn’t testing
Recommended Reading
If you want to learn more about testing in general, I recommend the following articles:
- Eradicating Non-Determinism in Tests, by Martin Fowler,
- Programmer Test Principles
- Responsibilities of the Test-Driven Developer in a Legacy Code Base, by James Grenning
- Mocks Aren’t Stubs by Martin Fowler
- Why TDD Isn’t Crap, by Hillel Wayne
- Unit Tests Aren’t Tests, by Hillel Wayne
- xUnit Test Patterns
For more information on testing concepts:
- Wikipedia: Mock Object
- Test Doubles, by Martin Fowler
- Self-Testing Code, by Martin Fowler
Did We Miss Anything?
If you have a testing resource that you love, leave us a comment below and we will add it to the list!
This blog is so amazing! Thank you! I was wondering if the notion of test pyramid (see https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid) would be worth mentioning here. Helps get a sense of the balance in test types for automation. I hope I didn’t miss an equivalent link 🙂
Thank you!
I haven’t seen the test pyramid yet. Definitely something to add. I’ll do some more research. Any other recommended resources on the subject?
Thanks for your sharring,very well writen! I’d like to add one more in your list – ZenTao (https://www.zentao.pm/), which is comes from China and very powerful! One of ZenTao/s function-ZTF is for all automated testing types. Recommend to you!