Reading Material

Embedded Systems Security Resources

Security is an area that we all need to focus on when we build our embedded systems. Here's our collection of resources on embedded systems security (and security in general). If you have recommended books or links that discuss embedded systems security, let us know!

Table of Contents:

Books

Some of these books may be slanted toward/against vendors. We recommend getting the security concepts without putting too much stock into the opinion of vendors.

Lectures

Phil Koopman has a collection of lectures discussing embedded systems security:

Articles

Primers

Here are articles with general introductions to embedded systems security concepts:

Pitfalls

These articles focus on common pitfalls with embedded device security:

General Security Articles

The authors of Embedded Systems Security published a series of articles on EDN:

Twitter Accounts

Here are some Twitter accounts to follow. Some accounts have an electronics/hardware bent, but others are more general InfoSec.

Acknowledgements

Embedded Systems Testing Resources

Updated: 20190302

Embedded Artistry was founded with the goal of creating reliable, safe, and well-tested embedded systems. The sad fact is that most embedded software that we've encountered is low-quality and untested. Perhaps this holds true for most of the software industry - the continual procession of hacks, flaws, and errors is discouraging.

We are focused on ways that we can all improve at our craft. Testing our systems, especially in an automated way, is a direct approach at identifying bugs early and improving our software quality.

We decided to make our list of testing resources publicly available. If you have any favorites which aren't listed here, leave us a comment!

Table of Contents:

  1. Unit testing
    1. James Grenning and TDD
    2. Matt Chernosky and Electron Vector
    3. Throw the Switch
    4. Unit Testing Frameworks
    5. Other Resources
  2. Debugger-based testing using metal.test
  3. Phil Koopman's lectures on embedded software quality and testing

Unit Testing

Most developers know they should be writing unit tests as they develop new features. If you're unfamiliar with the concept, unit testing focuses on testing the finest granularity of our software, the functions and modules, in an independent and automated manner. We want to ensure that the small pieces operate correctly before we combine them into larger cooperating modules. By testing at the finest granularity, we reduce the total number of test combinations needed to cover all possible logic states. By testing in an automated manner, we can ensure that any changes we make don't introduce unintended errors.

"A key advantage of well tested code is the ability to perform random acts of kindness to it. Tending to your code like a garden. Small improvements add up and compound. Without tests, it's hard to be confident in even seemingly inconsequential changes."
-Antonio Cangiano (@acangiano)

Unfortunately, at Embedded Artistry we’ve only worked on a handful of projects that perform unit testing. The primary reason for this is that many developers simply don’t know where to start when it comes to writing unit tests for embedded systems. The task feels so daunting and the schedule pressures are so strong that they tend to avoid unit testing all together.

James Grenning and TDD

James Grenning has put a tremendous amount of effort into teaching embedded systems developers how to adopt TDD. He published an embedded systems classic, Test-Driven Development for Embedded C and conducts TDD training seminars.

James has written extensively about TDD on his blog. Here are some of our favorite posts:

You can also watch these talks for an introduction into the how and why of TDD:

I took James Grenning's Remote TDD training webinar and wrote about my experience on the blog.

If you're interested in taking a training class, you can learn more on James's website:

Matt Chernosky and Electron Vector

Matt Chernosky, who runs the Electron Vector blog, is an invaluable resource for embedded unit testing and TDD. If you are having a hard time getting started with unit testing and TDD, Matt's articles provide a straightforward and accessible approach.

Here are some of our favorite articles from Matt's blog:

Matt published a free guide for using Ceedling for TDD. If you have experience with TDD but are looking to improve your Ceedling skills, check out his Ceedling Field Manual

Throw the Switch

Throw the Switch created the Ceedling, Unity, and Cmock unit testing suite. This is the trio of tools that Matt Chernosky uses and writes about.

Aside from creating testing tools, Throw the Switch maintains a library of test-related articles and a course called Unit Testing & Other Embedded Software Catalysts. Additional courses related to unit testing for embedded systems are being developed.

Unit Testing Frameworks

Listed below are frequently recommended unit testing frameworks for C and C++. There are more unit testing frameworks in existence than we can ever review, so again this list is not exhaustive. If nothing jumps out at you in this list, keep looking to find one that fits your team’s development style.

We already mentioned the Throw the Switch C unit testing frameworks:Unity, Cmock, and Ceedling.

Cmocka is the C unit testing framework we started with. Cmocka ships with built-in mock object support and operates similarly to Unity & CMock. The framework is built with C standard library functions and works well for embedded systems testing.

Catch appears to be the most popular C++ unit testing framework. Catch is a header-only library which supports C++11, C++14, and C++17.

Doctest is the unit test framework we use for EA’s C++ embedded framework project. Doctest is similar to Catch and is also header-only. Our favorite attribute of Doctest is that it keeps the test code alongside the implementation code. Doctest also enables you to write tests in headers, which Catch does not support.

GoogleTest is Google's C++ unit testing framework. GoogleTest is one of the few C++ frameworks with built-in mocking support.

CppUTest is a C++ test suite that was designed with embedded developers in mind. This framework is featured in James Grenning's book Test-Driven Development for Embedded C. C++ features within the framework are kept to a minimum enabling it to be used for both C and C++ unit testing. CppUTest has built-in support for mocking.

If you're interested in mock object support for C++, check out GoogleMock, Trompeloeil, and FakeIt. Each of these mocking frameworks can be integrated with the unit test frameworks mentioned above.

Other Resources

If you're brand-new to TDD, read through this walkthrough to get a sense of the approach:

Test Driven Development: Example Walkthrough

Steve Branam, who writes at Flink and Blink, has written posts on testing:

Steve recommended Jeff Langr's Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better as another resource for embedded C++ developers. .

Embedded Debugger-Based Testing

We always want to run as many tests as possible on a host PC when unit testing embedded systems code. However, we can’t test every aspect of our system on a host machine. Before shipping the final system, we need to evaluate the target compiler, issues which only present themselves on the target (e.g. endianness, timing), and the actual target hardware and hardware interfaces.

Metal.test is a framework which can help us with on-target testing. Klemens Morgenstern, an independent contractor and consultant, maintains this project. Metal.test enables automated execution of remote code using debugger hardware, such as J-LINK or ST-Link. The project supports gdb, and he has lldb support planned for a future release.

Metal.test features:

  • I/O Forwarding
  • Code Coverage
  • Unit testing
  • Call tracing
  • Profiling
  • Function Stubbing at link-time
  • Extensive documentation (in the GitHub repository Wiki)

Metal.test also includes a plugin system. While it's not essential, plugin support enables developers to extend the functionality to support any use case that a debugger can support.

Klemens is looking for feedback on metal.test. Don't hesitate to reach out with questions, issues, or other feedback.

Phil Koopman on Testing and Software Quality

Phil Koopman is a professor at Carnegie Mellon University. He also runs the Better Embedded Software blog, which is a must-read for embedded developers.

Phil has produced an immense and invaluable body of work, much of it focused on embedded software quality. The lecture notes for his embedded systems courses are available online, and he posts lecture videos on his Youtube channel.

Here's a selection of his lectures related to testing and software quality:

Did We Miss Anything?

If you have a testing resource that you love, leave us a comment below and we will add it to the list!

Change Log

  • 20190302:*
    • Added Phil Koopman lecture series on testing
    • Improved grammar
    • Improved table of contents

Matt Chernosky and Electron Vector

Matt Chernosky, who runs the Electron Vector blog, is an invaluable resource for embedded unit testing and TDD. If you are having a hard time getting started with unit testing and TDD, Matt's articles provide a straightforward and accessible approach.

Here are some of our favorite articles from Matt's blog:

Matt published a free guide for using Ceedling for TDD . If you are experienced with TDD but are looking to improve your Ceedling skills, check out his Ceedling Field Manual.

Throw the Switch

Throw the Switch created the Ceedling , Unity , and Cmock unit testing suite. This is the trio of tools that Matt Chernosky uses and writes about.

Aside from creating testing tools, Throw the Switch maintains a library of test-related articles and a course called Unit Testing & Other Embedded Software Catalysts . Additional courses related to unit testing for embedded systems are being developed.

Unit Testing Frameworks

Listed below are frequently recommended unit testing frameworks for C and C++. There are more unit testing frameworks in existence than we can ever review, so again this list is not exhaustive. If nothing jumps out at you in this list, keep looking to find one that fits your team’s development style.

We already mentioned the Throw the Switch C unit testing frameworks: Unity , Cmock , and Ceedling .

Cmocka is the C unit testing framework we started with. Cmocka ships with built-in mock object support and operates similarly to Unity & CMock. The framework is built with C standard library functions and works well for embedded systems testing.

Catch appears to be the most popular C++ unit testing framework. Catch is a header-only library which supports C++11, C++14, and C++17.

Doctest is the unit test framework we use for EA’s C++ embedded framework project. Doctest is similar to Catch and is also header-only. Our favorite attribute of Doctest is that it keeps the test code alongside the implementation code. Doctest also enables you to write tests in headers, which Catch does not support.

GoogleTest is Google's C++ unit testing framework. GoogleTest is one of the few C++ frameworks with built-in mocking support.

CppUTest is a C++ test suite that was designed with embedded developers in mind. This framework is featured in James Grenning's book _ Test-Driven Development for Embedded C _. C++ features within the framework are kept to a minimum enabling it to be used for both C and C++ unit testing. CppUTest has built-in support for mocking.

If you're interested in mock object support for C++, check out GoogleMock , Trompeloeil , and FakeIt . Each of these mocking frameworks can be integrated with the unit test frameworks mentioned above.

Other Resources

If you're brand-new to TDD, read through this walkthrough to get a sense of the approach:

Steve Branam, who writes at Flink and Blink , has written a few posts on testing:

Steve recommended Jeff Langr's Modern C++ Programming with Test-Driven Development: Code Better, Sleep Better as another resource for embedded C++ developers.

Embedded Debugger-Based Testing

We always want to run as many tests as possible on a host PC when unit testing embedded systems code. However, we can’t test every aspect of our system on a host machine. Before shipping the final system, we need to evaluate the target compiler, issues which only present themselves on the target (e.g. endianness, timing), and the actual target hardware and hardware interfaces.

Metal.test is a framework which can help us with on-target testing. This project is maintained by Klemens Morgenstern, an independent contractor and consultant. Metal.test enables automated execution of remote code using debugger hardware, such as J-LINK or ST-Link. The project currently supports gdb, and lldb support is planned for a future release.

Metal.test features:

  • I/O Forwarding
  • Code Coverage
  • Unit testing
  • Call tracing
  • Profiling
  • Function Stubbing at link-time
  • Extensive documentation (in the GitHub repository Wiki )

Metal.test also includes a plugin system. While it's not essential, plugin support enables developers to extend the functionality to support any use case that a debugger can support.

Klemens is looking for feedback on metal.test . Don't hesitate to reach out with questions, issues, or other feedback.

Phil Koopman on Testing and Software Quality

Phil Koopman is a professor at Carnegie Mellon University . He also runs the Better Embedded Software blog , which is a must-read for embedded developers.

Phil has produced an immense and invaluable body of work, much of it focused on embedded software quality. The lecture notes for his embedded systems courses are available online, and he regularly posts lecture videos on his Youtube channel .

Here's a selection of his lectures that are related to testing and software quality:

Did We Miss Anything?

If you have a testing resource that you love, leave us a comment below and we will add it to the list!

Related Articles

Timeless Laws of Software Development

I am always seeking the wisdom and insights of those who have spent decades working in software development. The experiences of those who came before us is a rich source of wisdom, information, and techniques.

Only a few problems in our field are truly new. Most of the solutions we seek have been written about time-and-time-again over the past 50 years. Rather than continually seeking new technology as the panacea to our problems, we should focus ourselves on applying the tried and tested basic principles of our field.

Given my point of view, it's no surprise that I was immediately drawn to a book titled Timeless Laws of Software Development.

The author, Jerry Fitzpatrick, is a software instructor and consultant who has worked in a variety of industries: biomedical, fitness, oil and gas, telecommunications, and manufacturing. Even more impressive for someone writing about the Timeless Laws of Software Development, Jerry was originally an electrical engineer. He worked with Bob Martin and James Grenning at Teradyne, where he developed the hardware for Teradyne's early voice response system.

Jerry has spent his career dealing with the same problems we are currently dealing with. It would be criminal not to steal and apply his hard-earned knowledge.

I recommend this invaluable book equally to developers, team leads, architects, and project managers.

Table of Contents:

  1. Structure of the Book
  2. The Timeless Laws
  3. What I Learned
  4. Selected Quotes
  5. Buy the Book

Structure of the Book

The book is short, weighing in at a total of 180 pages, including the appendices, glossary, and index. Do not be fooled by its small stature, for there is much wisdom packed into these pages.

Jerry opens with an introductory chapter and dedicates an entire chapter to each of his six Timeless Laws (discussed below). Each law is broken down into sub-axioms, paired with examples, and annotated with quotes and primary sources.

Aside from the always-useful glossary and index, Jerry ends the book with three appendices, each valuable in its own right:

  • "About Software Metrics", which covers metrics including lines of code, cyclomatic complexity, software size, and Jerry's own "ABC" metric
  • "Exploring Old Problems", which covers symptoms of the software crisis, the cost to develop software, project factors and struggles, software maintenance costs, superhuman developers, and software renovation.
  • "Redesigning a Procedure", where Jerry walks readers through a real-life refactoring exercise

"Exploring Old Problems" was an exemplary chapter. I highly recommended it to project managers and team leads.

My only real critique of the book is that the information is not partitioned in a way that makes it easily accessible to different roles - project managers may miss valuable lessons while glossing over programming details. Don't give in to the temptation to skip: each chapter has valuable advice no matter your role.

The Timeless Laws

Jerry proposes six Timeless Laws of software development:

  1. Plan before implementing
  2. Keep the program small
  3. Write clearly
  4. Prevent bugs
  5. Make the program robust
  6. Prevent excess coupling

At first glance, these six laws are so broadly stated that the natural reaction is, "Duh". Where the book shines is in the breakdown of these laws into sub-axioms and methods for achieving the intent of the law.

Breakdown of the Timeless Laws

  1. Plan before implementing
    1. Understand the requirements
    2. Reconcile conflicting requirements
    3. Check the feasibility of key requirements
    4. Convert assumptions to requirements
    5. Create a development plan
  2. Keep the program small
    1. Limit project features
    2. Avoid complicated designs
    3. Avoid needless concurrency
    4. Avoid repetition
    5. Avoid unnecessary code
    6. Minimize error logging
    7. Buy, don't build
    8. Strive for Reuse
  3. Write clearly
    1. Use names that denote purpose
    2. Use clear expressions
    3. Improve readability using whitespace
    4. Use suitable comments
    5. Use symmetry
    6. Postpone optimization
    7. Improve what you have written
  4. Prevent bugs
    1. Pace yourself
    2. Don't tolerate build warnings
    3. Manage Program Inputs
    4. Avoid using primitive types for physical quantities
    5. Reduce conditional logic
    6. Validity checks
    7. Context and polymorphism
    8. Compare floating point values correctly
  5. Make the program robust
    1. Don't let bugs accumulate
    2. Use assertions to expose bugs
    3. Design by contract
    4. Simplify exception handling
    5. Use automated testing
    6. Invite improvements
  6. Prevent excess coupling
    1. Discussion of coupling
    2. Flexibility
    3. Decoupling
    4. Abstractions (functional, data, OO)
    5. Use black boxes
    6. Prefer cohesive abstractions
    7. Minimize scope
    8. Create barriers to coupling
    9. Use atomic initialization
    10. Prefer immutable instances

What I Learned

I've regularly referred to this book over the past year. My hard-copy is dog-eared and many pages are covered in notes, circles, and arrows.

I've incorporated many aspects of the book into my development process. I've created checklists that I use for design reviews and code reviews, helping to ensure that I catch problems as early as possible. I've created additional documentation for my projects, as well as templates to facilitate ease of reuse.

Even experienced developers and teams can benefit from a review of this book. Some of the concepts may be familiar to you, but we all benefit from a refresher. There is also the chance that you will find one valuable gem to improve your practice, and isn't that worth the small price of a book?

The odds are high that you'll find more than one knowledge gem while reading Timeless Laws.

Here are some of the lessons I took away from the book:

  1. Create a development plan
  2. Avoid the "what if" game
  3. Logging is harmful
  4. Defensive programming is harmful
  5. Utilize symmetry in interface design

Create a Development Plan

We are all familiar with the lack of documentation for software projects. I'm repeatedly stunned by teams which provide no written guidance or setup instructions for new members. Jerry points out the importance of maintaining documentation:

Documentation is the only way to transfer knowledge without describing things in person.

One such method that I pulled from the book is the idea of the "Development Plan". The plan serves as a guide for developers working on the project. The plan describes the development tools, project, goals, and priorities.

As with all documentation, start simple and grow the development plan as new information becomes available or required. Rather than having a large document, it's easy to break the it up into smaller, standalone files. Having separate documents will help developers easily find the information they need. The development plan should be kept within the repository so developers can easily find and update it.

Topics to cover in your development plan include:

  • List of development priorities
  • Code organization
  • How to set up the development environment
  • Minimum requirements for hardware, OS, compute power, etc.
  • Glossary of project terms
  • Uniform strategy for bug prevention, detection, and repair
  • Uniform strategy for program robustness
  • Coding style guidelines (if applicable)
  • Programming languages to be used, and where they are used
  • Tools to be used for source control, builds, integration, testing, and deployment
  • High-level organization: projects, components, file locations, and naming conventions
  • High-level logical architecture: major sub-systems and frameworks

Development plans are most useful for new team members, since they can refer to the document and become productive without taking much time from other developers. However, your entire team will benefit from having a uniform set of guidelines that can be easily located and referenced.

Avoid the "What If" Game

Many of us, myself included, are guilty of participating in the "what if" game. The "what if" game is prevalent among developers, especially when new ideas are proposed. The easiest way to shoot a hole in a new idea is to ask a "what if" question: "This architecture looks ok, but what if we need to support 100,000,000 connections at once?"

The "what if" game is adversarial and can occur because:

  • Humans have a natural resistance to change
  • Some people enjoy showing off their knowledge
  • Some people enjoy being adversarial
  • The dissenter dislikes the person who proposed the idea
  • The dissenter does not want to take on additional work

"What if" questions are difficult to refute, as they are often irrational. We should always account for realistic possibilities, but objections should be considered only if the person can explain why the proposal is disruptive now or is going to be disruptive in the future.

Aside from keeping conversations focused on realistic possibilities, we can mitigate the ability to ask "what if" with clear and well-defined requirements.

Logging is Harmful

I have been a long-time proponent of error logging, and I’ve written many embedded logging libraries over the past decade.

While I initially was skeptical of Fitzpatrick’s attitude toward error logging, I started paying closer attention to the log files I was working with as well as the use of logging in my own code. I noticed the points that Jerry highlighted: my code was cluttered, logs were increasingly useless, and it was always a struggle to remove outdated logging statements.

You can read more about my thoughts on error logging in my article: The Dark Side of Error Logging.

Defensive Programming is Harmful

Somewhere along the way in my career, the idea of defensive programming was drilled into me. Many of my old libraries and programs are layered with unnecessary conditional statements and error-code returns. These checks contribute to code bloat, since they are often repeated at multiple levels in the stack.

Jerry points out that in conventional product design, designs are based on working parts, not defective ones. As such, designing our software systems based on the assumption that all modules are potentially defective leads us down the path of over-engineering.

Trust lies at the heart of defensive programming. If no module can be trusted, then defensive programming is imperative. If all modules can be trusted, then defensive programming is irrelevant.

Like conventional products, software should be based on working parts, not defective ones. Modules should be presumed to work until proven otherwise. This is not to say that we don't do any form of checking: inputs from outside of the program need to be validated.

Assertions and contracts should be used to enforce preconditions and postconditions. Creating hard failure points helps us to catch bugs as quickly as possible. Modules inside of the system should be trusted to do their job and to enforce their own requirements.

Since I've transitioned toward the design-by-contract style, my code is much smaller and easier to read.

Utilize Symmetry in Interface Design

Using symmetry in interface design is one of those points that seemed obvious on the surface. Upon further inspection, I found I regularly violated symmetry rules in my interfaces.

Symmetry helps us to manage the complexity of our programs and reduce the amount of knowledge we need to keep in mind at once. Since we have existing associations with naming pairs, we can easily predict function names without needing to look them up.

Universal naming pairs should be used in public interfaces whenever possible:

  • on/off
  • start/stop
  • enable/disable
  • up/down
  • left/right
  • get/set
  • empty/full
  • push/pop
  • create/destroy

Our APIs should also be written in a consistent manner:

  • Motor::Start() / Motor::Stop()
  • motor_start() / motor_stop()
  • StartMotor() / StopMotor()

Avoid creating (and fix!) inconsistent APIs:

  • Motor::Start() / Motor::disable()
  • startMotor / stop_motor
  • start_motor / Stop_motor

Naming symmetry may be obvious, but where I am most guilty is in parameter order symmetry. Our procedures should utilize the same parameter ordering rules whenever possible.

For example, consider the C standard library functions defined in string.h. In all but one procedure (strlen), the first parameter is the destination string, and the second parameter is the source string. The parameter order also matches the normal assignment order semantics (dest = src).

The standard library isn't the holy grail of symmetry, however. The stdio.h header showcases some bad symmetry by changing the location of the FILE pointer:

int fprintf ( FILE * stream, const char * format, ... );
int fscanf ( FILE * stream, const char * format, ... );

// Better design: FILE is first!
int fputs ( const char * str, FILE * stream );
char * fgets ( char * str, int num, FILE * stream );

Keeping symmetry in mind will improve the interfaces we create.

Selected Quotes

I pulled hundreds of quotes from this book, and you will be seeing many of them pop up on our Twitter Feed over the next year. A small selection of my highlights are included below.

Any quotes without attribution come directly from Jerry.

Intentionally hiding a bug is the greatest sin a developer can commit.

Failure is de rigueur in our industry. Odds are, you're working on a project that will fail right now.
-- Jeff Atwood, How to Stop Sucking and Be Awesome

Writing specs is like flossing: everybody agrees that it's a good thing, but nobody does.
-- Joel Spolsky

Documentation is the only way to transfer knowledge without describing things in person.

Robustness must be a goal and up front priority.

Disorder is the natural state of all things. Software tends to get larger and more complicated unless the developers push back and make it smaller and simpler. If the developers don't push back, the battle against growth is lost by default.

YAGNI (You ain't gonna need it):
Always implement things when you actually need them, never when you just foresee that you need them. The best way to implement code quickly is to implement less of it. The best way to have fewer bugs is to implement less code.

-- Ron Jeffries

Most developers write code that reflects their immediate thoughts, but never return to make it smaller or clearer.

The answer is to clear our heads of clutter. Clear thinking becomes clear writing; one can't exist without the other.
-- William Zinsser

Plan for tomorrow but implement only for today.

Code that expresses its purpose clearly - without surprises - is easier to understand and less likely to contain bugs.

Most developers realize that excess coupling is harmful but they don't resist it aggressively enough. Believe me: if you don't manage coupling, coupling will manage you.

Few people realize how badly they write.
-- William Zinsser

To help prevent bugs, concurrency should only be used when needed. When it is needed, the design and implementation should be handled carefully.

Sometimes problems are poorly understood until a solution is implemented and found lacking. For this reason, it's often best to implement a basic solution before attempting a more complete and complicated one. Adequate solution are usually less costly than optimal ones.

I've worked with many developers who didn't seem to grasp the incredible speed at which program instructions execute. They worried about things that would have a tiny effect on performance or efficiency. They should have been worried about bug prevention and better-written code.

Most sponsors would rather have a stable program delivered on-time than a slightly faster and more efficient program delivered late.

It's better to implement features directly and clearly, then optimize any that affect users negatively.

Efficiency and performance are only problems if the requirements haven't been met. Optimization usually reduces source code clarity, so it isn't justified for small gains in efficiency or performance. Our first priorities should be correctness, clarity, and modest flexibility.

Implementation is necessarily incremental, but a good architecture is usually holistic. It requires a thorough understanding of all requirements.

Buy the Book

If you are interested in purchasing Timeless Laws of Software Development, you can support Embedded Artistry by using our Amazon affiliate link:

Related Posts