Embedded Artistry was originally founded with the goal of building reliable, safe, and well-tested embedded systems for clients. The sad fact is that most embedded software that we’ve encountered is low-quality and untested. Perhaps this holds true for most of the software industry – the continual procession of hacks, flaws, and errors is discouraging.
Testing our systems, especially in an automated way, is a direct approach at identifying bugs early and improving our software quality.
Table of Contents:
- Testing Concepts
- Course Lessons
- Lectures
- Articles From Around the Web
- Articles From Embedded Artistry
Testing Concepts
All systems should have a formal test plan document that provides a reproducible set of tests and criteria that are used to evaluate a system’s behavior.
There are many different categories of testing. They are often differentiated by scope and purpose. Common testing taxonomies include:
- Unit Testing
- Integration Testing
- Smoke Testing
- Automated Product Testing, involving automated stress tests for a product such as reboot cycling, repeated software updates, downgrading software automatically, etc.
- Fuzzing
- Quality Assurance
- Dogfooding, where an organization tests its own product
- Power-on Self Test, a suite of internal tests run during system startup to check overall health
- Memory Testing, used to ensure that memory is operating correctly (often performed as part of a power-on self test)
- Manufacturing Test, which are tests to ensure that products are assembled and operating properly before sending them to customers
There are common descriptions associated with testing taxonomies, such as:
- Black Box, which involves testing a system or component from the outside without looking inside
- White Box, which
- Host-only, or simulated
- On-target, which
- Hardware-in-the-loop, which
Other core concepts discussed in relation to testing include:
- Test Coverage
- Code Coverage
There are development processes that involve writing tests before writing code, including:
- Behavior Driven Development
- Test Driven Development
- Acceptance Test Driven Development
Testing Doesn’t Mean You Are Safe
We need a plan for proving that our software works safely. Testing is necessary, but it is not enough.
Phil Koopman recently wrote about a concept he calls The Insufficient Testing Pitfall:
Testing less than the target failure rate doesn’t prove you are safe. In fact you probably need to test for about 10x the target failure rate to be reasonably sure you’ve met it. For life critical systems this means too much testing to be feasible.
Course Lessons
Lectures
Phil Koopman is a professor at Carnegie Mellon University and author of the Better Embedded Software blog. Here’s a selection of his lectures related to testing:
- Embedded Software Testing Overview
- Unit Testing (slides only)
- Integration Testing (slides only)
- Testing Quality (slides)
- System Level Testing (slides)
- SQA isn’t testing
Articles From Around the Web
Steve Branam, who writes at Flink and Blink, provides a pragmatic take on why we should test in “Testing is how you avoid looking stupid”.
If you want to learn more about testing in general, I recommend the following articles:
- Eradicating Non-Determinism in Tests, by Martin Fowler
- Programmer Test Principles
- Mocks Aren’t Stubs by Martin Fowler
For more information on testing concepts:
- Metamorphic Testing by Hillel Wayne
- Wikipedia: Mock Object
- Test Doubles, by Martin Fowler
- Self-Testing Code, by Martin Fowler