December 2018

Welcome to the December 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Update on Freestanding C++
  • A new C++ proposal: Extensions of Scoped Enumerations
  • Code cleanup sprints
  • Bloomberg's "Big Hack" follow-up
  • Embedded news from around the web

Update on Freestanding C++

Ben Craig has been heading up the Freestanding C++ effort, which we discussed in the June 2018 and August 2018 editions of the newsletter. We are excited about a Freestanding C++ subset, as we frequently write C++ code destined for embedded devices without an OS. For most embedded platforms, we have to hack the language apart to avoid problematic areas. An official subset would be a welcome change.

In November, Ben traveled to the WG21 San Diego meeting and posted a detailed update on Reddit. The Freestanding C++ proposal continues to be discussed and refined, and WG21 San Diego included a discussion around how Freestanding C++ might work within the scope of C++ modules. Ben also reports that companies are beginning to adopt the new terminology proposed in his Freestanding proposal.

These are the latest Freestanding-related papers:

Extensions of Scoped Enumerations Proposal

We’ve been working on our framework over the past few months and have found ourselves frequently wishing for the ability to extend enumerations. We’d like to provide baseline definitions in our framework, but allow users to extend those definitions according to their specific needs. Sadly, this is not possible with C++. There are multiple proposed workarounds, but we haven’t liked anything we’ve tried thus far.

A new proposal popped up in the SG14 discussion group which recently caught our eye: Extensions of Scoped Enumerations. This paper describes improvements to scoped enumerations (enum class), which includes the ability to extend an enum class definition through an inheritance-like mechanism. The proposal's author also released a source repository which has forked clang to demonstrate the changes.

This proposal is still in the formative stages, but we are looking forward to developments in this area.

Code Cleanup Sprints

In Timeless Laws of Software Development, Jerry Fitzpatrick talks about the need to continually revise our code. Writers never dream of submitting their first draft; my own articles and newsletters go through at least two rounds of editing. However, as developers we are rarely that diligent. Most of our code is published as a first draft and not revisited unless an issue arises that forces our hand.

After reading Timeless Laws, we instituted a monthly "code cleanup sprint" at Embedded Artistry. On the last Friday of each month, we spend the entire workday poking around our repositories, refactoring, fixing TODOs, improving comments, and making other small codenimprovements.

Every software project is a massive investment of time, energy, and knowledge. Like a garden, without proper tending, it grows out of control. The more we neglect our maintenance efforts, the more work it takes to beautify and restore order to our code base. Now we look forward to these cleanup days and the deep sense of satisfaction that comes from taking the time to improve our project’s overall quality.

We hope that your teams can also dedicate a full day or more each month for tending to your project’s code base.

Bloomberg's "Big Hack" Followup

Another month has gone by and Bloomberg still stands by their Big Hack story, which we discussed in last month's newsletter.

Likewise, the companies named in the story still deny the claims. Both Amazon and Apple have pulled ads from Bloomberg's website in response to the article, and Apple has banned Bloomberg reporters from their next event. SuperMicro told Congress that they found no spy chips during their internal investigations. They claim that the “test processes at every step are designed to alert us to any discrepancies from our base design”. This seems to be an exaggerated claim based on our experiences with electronics manufacturing.

The Washington Post, owned by Jeff Bezos, continues to fire shots at Bloomberg. The article claims that another Bloomberg reporter is performing a separate analysis to try to confirm the details of the story. Perhaps Bloomberg is close to crumbling?

On the technical front, Serve the Home describes why the Big Hack is improbable in this excellent article.

As we mentioned last month, we have to doubt the validity of the story given the lack of corroborating evidence or confirmation from any named parties.

Around the Web

The security flaws keep rolling in:

The US indicted Taiwanese foundry UMC and Chinese DRAM maker Fujian Jinhua with conspiracy to steal IP from Micron. Another article, "Knowledge Transfer, or IP Theft?", provides a deeper analysis at the situation.

Mohammad Afaneh updated his excellent introduction to Bluetooth GATT. Mohammad's website is a valuable resource for anyone working on a Bluetooth device.

IEEE Spectrum published "The Case Against Quantum Computing", which provides a sobering look at the difficulties underlying the technology.

EETimes collected links to Aspencore's Road to Mars Special Project. Two interesting articles related to embedded systems:

Steve Branam revived an excellent interview from Jack Ganssle's archives, where Jack interviews James Grenning about TDD:

Website Updates

We made the following updates to the Embedded Artistry website:

New Articles

We published the following articles in November:

  1. Musings on Supply Chain Vulnerability in Light of The Big Hack
  2. A Look at Ten Hardware Startup Blunders, Pt. 1: Process
  3. A Look at Ten Hardware Startup Blunders, Pt. 2: Schedule and Focus
  4. Missing /usr/include after updating to OSX 10.14? Try this fix
  5. A Look at Ten Hardware Startup Blunders, Pt. 3: Team

These were our most popular articles in November:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Kick off a CI Build with GitHub Push Notifications
  8. Jenkins: Running Steps as sudo
  9. Implementing Malloc: First-fit Free List
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!


November 2018: The Big Hack

Welcome to the November 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The Big Hack story and its ensuing drama
  • A request for modern Embedded Linux resources
  • Embedded articles from around the web
  • Website updates

The Big Hack

At the beginning of this month, Bloomberg dropped a bomb by publishing "The Big Hack: How China Used a Tiny Chip to Infiltrate America's Top Companies". They followed the initial story with details about the software side of the attack. The Bloomberg article is a thrilling and dramatic read which has stirred up quite a bit of controversy.

If you haven't read the original story, here's a brief summary:

Between 2014-2015, Supermicro server motherboards had a small IC inserted onto the PCB, supposedly connected to the baseboard management controller (BMC). The chip allowed attackers to alter “the server’s Core OS so it could accept modifications and contact attacker-controlled computers for further instructions/code”. Supermicro subcontractors installed these chips at the behest of the People’s Liberation Army (PLA) of China. The software side of the picture involves two vectors: shipping Supermicro servers with outdated firmware that contained known security vulnerabilities, and releasing firmware updates with new vulnerabilities which would be installed after the boards were already in customers’ hands.

Unsurprisingly, the story has been vehemently denied by all parties involved:

Bloomberg was transparent about the companies' denials and published them on their website. Bloomberg then doubled down on the Big Hack by publishing a follow-up article stating that new evidence of hacked Supermicro hardware was found in a (unnamed) US telecom. Unlike the "tiny chip" on the motherboard, the reported hack involved an implant built into the Ethernet connector.

The Big Hack and the ensuing wave of denials have read like a fictional spy thriller. Following the commentary surrounding the story has been fun and enlightening:

  • Joe FitzPatrick at Securing Hardware, one of the sources named in the story, shared his thoughts around hardware implants, how he'd execute such an attack, and how compromising the BMC firmware would be a much simpler approach than implementing the hardware hack.
  • Bloomberg published the story a month after the NASDAQ delisted Supermicro.
  • Supermicro shares plunged ~50% following the first announcement.
    • 20+ people made suspicious short positions and sale orders just before the story broke, leading to calls for an SEC investigations and suspicion that this is a Supermicro hit piece.
  • Matt Levine speculated on whether securities fraud would apply if companies falsely denied the story.
  • Risky Business published a podcast digging into the story and interviewed Joe FitzPatrick, who said he feels “uneasy” about the story as published.
    • Interestingly, Vice states that Joe FitzPatrick "told Motherboard in an online chat he was 'not surprised to see an example finally.'" This is an odd statement for someone claiming the story made them feel uneasy.
  • Serve the Home analyzed the story and the BMC theory.
  • Vice dug into supply chain vulnerabilities and included this interesting quote:
    • “There are two possible stories here,” Matthew Green, associate professor at Johns Hopkins University, tweeted about the attack. “One is that there was an attack. The other is that a large swath of the National Security establishment is promoting the idea that there was an attack. Pick your poison.”
  • Sup China takes a similar tack as Matthew Green and postulates that government sources may be trying to generate or amply mistrust of China.
  • One Bloomberg opinion piece relates the hack to the growing conflicts between the US and China. Another opinion piece postulates on how hopelessly vulnerable global supply chains are - further stirring the pot in regards to China's business practices.

Now that a month has passed, we seem to be at an impasse. The lack of evidence of hacked hardware, a list of affected SKUs, or confirmation by other agencies casts doubt on Bloomberg’s story. The companies named continue to deny the story, and Bloomberg will not make a retraction, name sources, or provide further details. Other news agencies have been unable to independently confirm the story.

Regardless of whether the story is true or false, it highlights the vulnerability of our modern technological supply chains. I published my musings on modern supply chains and the feasibility of hardware tampering on the website.

A Request for Modern Embedded Linux Resources

I've received emails asking for recommendations for up-to-date Embedded Linux resources. We primarily focus on writing software for microcontrollers, so I have little experience with Embedded Linux. Most of the reference material that I've seen is at least a decade old or focuses on Raspberry Pi development using python libraries.

If you're an Embedded Linux developer who writes drivers or other low-level code, I'd love to hear about your favorite books, websites, and courses. Where do you turn to when you have a problem? What training do you take to improve your skills? What tools do you use most often?

You can respond directly to this email or send us a message on Twitter.

Around the Web

Another day, another security flaw announcement, this time for Amazon FreeRTOS.

Phil Koopman highlighted the cost of producing highly safety-critical software.

In the December 2017 edition of the newsletter we covered advances in chip design techniques. Read this IEEE article for another new chip design technique: Through-Silicon Transistors Could Make Stacking Chips Smarter.

Meeting Embedded shared a short interview with Dan Saks, a champion of embedded C++ development.

Mohammad Afaneh at Novel Bits continued publishing his Bluetooth Mesh tutorial series:

Jack Ganssle finished publishing "Top 10 Reasons Embedded Projects Get Into Trouble". He finished his serious with two of the most common problems I see: "Quality Gets Lip Service" and "Unrealistic Schedules".

Website Updates

The Glossary received another extensive update focused on manufacturing and supply chain terminology.

We updated our Technology Radar to include the Espressif ESP32 chipset. Saket Vora introduced us to this chip, which features 512kB SRAM, qSPI supporting 16Mbit external flash, low deep sleep current, and a superb datasheet. We're evaluating it for use with our embedded framework.

New Articles

We published the following articles in October:

We also updated the following articles with new content:

These were our most popular articles in October:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Jenkins: Kick off a CI Build with GitHub Push Notifications
  9. Migrating from C to C++: NULL vs nullptr
  10. Implementing Malloc: First-fit Free List

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next email, check out the website or follow us on Twitter.

Happy hacking!


October 2018: EA Framework Announcement & More Embedded Libraries

Welcome to the October 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Embedded Artistry's C++ Framework Project
  • Three more projects embedded developers should be aware of
  • Embedded articles from around the web
  • Website updates

Embedded Artistry Framework Project

Over the past two years in San Francisco, I’ve had the opportunity to consult with dozens of companies and to work on a wide variety of embedded products. Most teams and projects seem to struggle with the same issues, especially with hiring firmware engineers and keeping projects on schedule.

Why is everyone running into the same roadblocks?

In order to find solutions, I've been immersing myself in the software engineering corpus, reflecting on the coupling between firmware and hardware, learning software architecture, practicing Test Driven Development, and researching embedded libraries and frameworks that can help firmware developers move faster.

What I’ve noticed is that the software industry has learned a lot of lessons over the past fifty years, but we’ve failed to incorporate many of them in the embedded community. We are building products in less time, with minimal teams, and generally using poorly tested vendor SDKs. We aren’t decoupling our software from the underlying hardware. Many embedded teams still don’t utilize modern development practices such as static analysis, unit testing, and continuous integration.

What if we actually leveraged the lessons we’ve learned as an industry? What if we built our embedded software in such a way that we minimized coupling with the underlying hardware and RTOS?

  • We could begin developing and testing our embedded software before we have hardware in house
  • We could write device drivers on our host machine (talking to real parts) and port them over to our target hardware with full confidence that they will work
  • We could defend against the inevitable changes in components and processors that trigger software rewrites and massive schedule delays
  • We could leverage proven libraries and frameworks to save time and avoid reinventing any wheels
  • We could automate our testing and check our code often for preventable bugs

Embedded Artistry is building an embedded C++ framework which will help us accomplish these goals. We want to provide embedded teams with a solid foundation that incorporates our lessons learned and supports modern development practices.

Our primary goal is to support decoupling firmware from hardware at various levels of abstraction (platform, circuit board, processor). Rather than aim for universal abstractions and interfaces, we want to provide a minimal foundation which teams can expand to meet their product’s specific goals and requirements.

We are currently building our basic framework structures on a host machine to ensure we can develop software off-target. Later this fall we’ll bring the framework up on a couple of common embedded platforms.

I’ll be sharing more about the framework as it evolves over the coming months. In the mean time, I’d love to hear from you if you think your team could benefit from a system like this.

Three More Useful Embedded Projects

I always have my eyes open for interesting embedded projects. I want to share three more projects that I've discovered, two of which are being actively used in our framework:

  • Embedded Template Library
  • mpaland/printf
  • embxx

For other embedded projects, libraries, and frameworks which we've shared, check out the following editions:

Embedded Template Library

A common concern with using C++ and the STL on embedded systems is the reliance on exceptions and dynamic memory allocation. For many embedded systems, use of these features is disallowed. Working around this limitation is possible, but often more difficult than project schedules allow for.

John Wellbelove of Aster Consulting Ltd created the Embedded Template Library (ETL) to address these issues. The ETL provides fixed-capacity STL container alternatives and requires no dynamic memory allocation, no RTTI, and no exceptions. The ETL also provides additional container types, helpful utilities, algorithms (such as CRC), reusable framework components (FSM, state chart, scheduler, callback timer), and an abundance of documentation.

If your embedded compiler doesn't support modern C++ standards, you're in luck - the library is compatible with C++03, and many of the C++11 features have been backported through the ETL.

The ETL is an becoming an essential tool for our embedded C++ development. We've adopted the ETL in our current framework project to take advantage of fixed-capacity containers and other useful features.


printf is one of those commands that every developer relies on. However, using printf on many embedded systems can be a challenge due to the size of the library (20kB+ is not uncommon). Perhaps you do not want to link with libc for your platform, or the implementation you're using requires you to implement esoteric system functions for each platform.

The mapaland/printf implementation addresses these problems. The implementation is ~600 lines and has no external dependencies. The library is reentrant and all of the useful flags (including floating point and precision/width specifiers) are supported. All you have to do to get up and running on your platform is define a _putchar() function which outputs a character over your chosen communication bus (UART, USB, BLE, etc.).

We've integrated mapaland/printf into our embeddedartistry/libc and are actively using it in our framework project.


If you're working with C++ on embedded systems, chances are that you've come across the Practical Guide to Bare Metal C++. This free e-book is an excellent starting point for applying C++ to embedded systems, and the author walks through the creation of many useful embedded systems constructs.

The concepts presented in the book are fleshed out in the companion embxx library. embxx is written in C++11 and targeted for bare metal and embedded Linux systems. The library doesn't use RTTI or exceptions, and most of the constructs presented in the library utilize static memory allocation.

Sadly, the project is no longer active, but it still serves as an excellent resource for embedded C++ developers. Many of the utilities, such as StaticFunction, StaticQueue, and StaticPoolAllocator, are always useful for embedded systems. The project also serves as a great reference for how we can utilize template meta-programming and static memory allocation schemes when writing embedded C++ software.

Around the Web

Steve Branam at Flink And Blink published "So You Want To Be An Embedded Systems Developer". This is an excellent article for those just starting out in the embedded systems field.

Matt Chernosky at Electron Vector shared 7 tips for adding unit tests to existing firmware. Start with Matt's tips if you need inspiration for adding tests to your existing project.

Mohammad Afaneh at Novel Bits published a four-part tutorial on Bluetooth Mesh. If you want to learn about Bluetooth Mesh, start with this series.

Mohammad also shared an interesting white paper in his weekly Bluetooth newsletter: Bluetooth Angle Estimation for Real-Time Locationing. The paper discusses using Bluetooth Angle of Arrival and Angle of Departure for indoor locationing.

Jack Ganssle continues to publish his "Top 10 Reasons Embedded Projects Get Into Trouble". We've worked our way to Number 3:

Website Updates

Our Glossary page has been expanded. If you notice any terms or acronyms that are missing, let us know!

We've restructured our Libraries page and added more of our own open-source projects to the list.

Our Technology Radar has been updated:

  • We are now using TDD and recommend it for adoption
  • We are recommending the Embedded Template Library for adoption after completing our trial period
  • We are experimenting with automated clang-tidy checks on our framework project
  • We have moved away from Doctest and are now trying out Catch for our C++ unit testing because it supports -fno-exceptions

New Articles

The following article was published on our website in September:

These were our most popular articles in September:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Migrating from C to C++: NULL vs nullptr
  9. Implementing an Asynchronous Dispatch Queue
  10. Jenkins: Kick off a CI Build with GitHub Push Notifications

Thanks for Reading!

September 2018: Testing Embedded Systems

Welcome to the September 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Testing Embedded Systems
  • Embedded articles from around the web
  • Embedded Artistry website updates

Testing Embedded Systems

Embedded Artistry was founded with the goal of creating reliable, safe, and well-tested embedded systems. The sad fact is that most embedded software that we've encountered is low-quality and untested. Perhaps this holds true for most of the software industry - the continued procession of hacks, flaws, and errors is discouraging.

We can talk about why this is all day long, instead we want to focus on ways that we can all improve at our craft.

We'll cover three facets of testing embedded systems:

  1. Unit testing
  2. Debugger-based testing using metal.test
  3. Phil Koopman's lectures on embedded software quality and testing

This is not an exhaustive list, and there is much more to say about software quality that we’ll continue to cover in future articles and newsletters. For today, focusing on testing is a great place to start.

Unit Testing

Most developers know they should be writing unit tests as they develop new features. If you're unfamiliar with the concept, unit testing focuses on testing the finest granularity of our software, the functions and modules, in an independent and automated manner. We want to ensure that the small pieces operate correctly before we combine them into larger cooperating modules. By testing at the finest granularity, we reduce the total number of test combinations that are needed to cover all possible logic states. By testing in an automated manner, we can ensure that any changes we make don't introduce unintended errors.

"A key advantage of well tested code is the ability to perform random acts of kindness to it. Tending to your code like a garden. Small improvements add up and compound. Without tests, it's hard to be confident in even seemingly inconsequential changes." -Antonio Cangiano (@acangiano)

Unfortunately, at Embedded Artistry we’ve only worked on a handful of projects that perform unit testing.

The primary reason for this is that many developers simply don’t know where to start when it comes to writing unit tests for embedded systems. The task feels so daunting and the schedule pressures are so strong that they tend to avoid unit testing all together.

After receiving a request for more resources on Test Driven Design (TDD) with embedded systems, we decided to share our favorite unit testing & TDD resources.

James Grenning and TDD

James Grenning has put a tremendous amount of effort into teaching embedded systems developers how to adopt TDD. He published an embedded systems classic, Test-Driven Development for Embedded C, and regularly conducts TDD training seminars.

James has written extensively about TDD on his blog. Here are some of our favorite posts:

You can also watch these talks for an introduction into the how and why of TDD:

Phillip is signed up to take James Grenning's TDD webinar, which begins today. If you're interested in taking a training class, look into these links:

Matt Chernosky and Electron Vector

Matt Chernosky, who runs the Electron Vector blog, is an invaluable resource for embedded unit testing and TDD. If you are having a hard time getting started with unit testing and TDD, Matt's articles provide a straightforward and accessible approach.

Here are some of our favorite articles from Matt's blog:

Matt published a free guide for using Ceedling for TDD. If you are experienced with TDD but are looking to improve your Ceedling skills, check out his Ceedling Field Manual.

Throw the Switch

Throw the Switch created the Ceedling, Unity, and Cmock unit testing suite. This is the trio of tools that Matt Chernosky uses and writes about.

Aside from creating testing tools, Throw the Switch maintains a library of test-related articles and a course called Unit Testing & Other Embedded Software Catalysts. Additional courses related to unit testing for embedded systems are being developed.

Unit Testing Frameworks

Listed below are frequently recommended unit testing frameworks for C and C++. There are more unit testing frameworks in existence than we can ever review, so again this list is not exhaustive. If nothing jumps out at you in this list, keep looking to find one that fits your team’s development style.

We previously covered the Throw the Switch C unit testing frameworks: Unity, Cmock, and Ceedling.

Cmocka is the C unit testing framework we started with. Cmocka ships with built-in mock object support and operates similarly to Unity & CMock. The framework is built with C standard library functions and works well for embedded systems testing.

Catch appears to be the most popular C++ unit testing framework. Catch is a header-only library which supports C++11, C++14, and C++17.

Doctest is the unit test framework we use for EA’s C++ embedded framework project. Doctest is similar to Catch and is also header-only. Our favorite attribute of Doctest is that it keeps the test code alongside the implementation code. Doctest also enables you to write tests in headers, which Catch does not support.

GoogleTest is Google's C++ unit testing framework. GoogleTest is one of the few C++ frameworks with built-in mocking support.

CppUTest is a C++ test suite that was designed with embedded developers in mind. This framework is featured in James Grenning's book Test-Driven Development for Embedded C. C++ features within the framework are kept to a minimum enabling it to be used for both C and C++ unit testing.

If you're interested in mock object support for C++, check out GoogleMock, Trompeloeil, and FakeIt. Each of these mocking frameworks can be integrated with the unit test frameworks mentioned above.

Embedded Debugger-Based Testing

We always want to run as many tests as possible on a host PC when unit testing embedded systems code. However, we can’t test every aspect of our system on a host machine. Before shipping the final system, we need to evaluate the target compiler, issues which only present themselves on the target (e.g. endianness, timing), and the actual target hardware and hardware interfaces.

Metal.test is a framework which can help us with on-target testing. This project is maintained by Klemens Morgenstern, an independent contractor and consultant. Metal.test enables automated execution of remote code using debugger hardware, such as J-LINK or ST-Link. The project currently supports gdb, and lldb support is planned for a future release.

Metal.test features:

  • I/O Forwarding
  • Code Coverage
  • Unit testing
  • Call tracing
  • Profiling
  • Function Stubbing at link-time
  • Extensive documentation (in the GitHub repository Wiki)

Metal.test also includes a plugin system. While it's not essential, plugin support enables developers to extend the functionality to support any use case that a debugger can support.

Klemens is looking for feedback on metal.test. Don't hesitate to reach out with questions, issues, or other feedback.

Phil Koopman on Testing & Software Quality

Phil Koopman is a professor at Carnegie Mellon University. He also runs the Better Embedded Software blog, which is a must-read for embedded developers.

Phil has produced an immense and invaluable body of work, much of it focused on embedded software quality. The lecture notes for his embedded systems courses are available online, and he regularly posts lecture videos on his Youtube channel.

Here's a selection of his lectures that are related to testing and software quality:

Around the Web

Mohammad Afaneh has finished his Introduction to Bluetooth Low Energy eBook. If you're working with Bluetooth and BLE, Mohammad is the person to follow.

NAND makers are striving for larger storage capacities. There are announcements of 512 Gbit, 1 Tbit, 1.33 Tbit capacities. SK Hynix is also working on a package with a whopping 8 Tbit capacity.

Jack Ganssle has started a blog at the end of July. Jack is also publishing a weekly series on which expand upon his "Top 10 Reasons Embedded Projects Get Into Trouble". So far, he's worked his way up to number 7:

You should also check out Phil Koopman’s slides for Avoiding the Top 43 Embedded Software Risks. How many risks has your team taken on?

Website Updates

We've added a list of our favorite software-related books to the Support page. If you're looking for a new programming or architecture book, you can support the website and newsletter by using the Amazon links found there.

Our popular Circular Buffers in C/C++ article was re-written based on learnings and feedback received over the past year.

The Embedded Artistry Technology Radar has received its first update with new techniques, libraries, and tools.

Three of our Resource pages have a brand-new look:

Our Glossary is always expanding and has been updated with new terms.

New Articles

These articles were published on our website in August:

These were our most popular articles in August:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Jenkins: Running Steps as sudo
  7. C++ Casting, or: "Oh No, They Broke Malloc!"
  8. Implementing an Asynchronous Dispatch Queue
  9. Migrating from C to C++: NULL vs nullptr
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter.

Happy hacking!


August 2018: Spectre Returns, Freestanding C++, Technology Radar

Welcome to the August 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • More Speculative Execution attacks: SpectreRSB and NetSpectre
  • The new version of the Freestanding C++ proposal
  • The Embedded Artistry Technology Radar
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Speculative Execution Attacks

Earlier this year we covered Spectre and Meltdown, two speculative execution vulnerabilities that affect a majority of the world's processors. In June, there was another announcement regarding an Intel speculative execution vulnerability dubbed lazy floating-point state restore.

The trend continues, and two more speculative execution attack vectors have come to light in the past few weeks:

  • SpectreRSB, which exploits speculative execution of the return stack buffer
  • NetSpectre, a remote attack which does not require attacker-controlled code to run on the victim's machine

Exploring these speculative execution vulnerabilities is a great way for embedded developers to learn the details about how processors work.


SpectreRSB is a speculative execution vulnerability which exploits the return stack buffer (RSB). The RSB is a structure similar to the branch predictor which is used to predict return addresses. When a call instruction is issued, the RSB pushes the return address onto an internal hardware stack.

SpectreRSB has six variants, but the basic flow is as follows:

  1. Context switch to the attacker:
    • The attacker flushes shared address entries (i.e. flush reload)
    • The attacker pollutes the RSB with the target address of a payload gadget in the victim’s address space
  2. The attacker yields the CPU back to the victim
  3. The victim eventually executes a return
    • Speculative execution occurs at the address which the attacker injected into RSB
  4. Control switches back to the attacker, data is leaked

The authors of the paper claim that the existing mitigations against Spectre variant 1 do not protect against SpectreRSB. Intel claims that SpectreRSB is related to branch target injections which have already been mitigated. On some processors, an RSB refilling patch is available that interferes with SpectreRSB’s attack mechanism.

Intel, AMD, and ARM processors are affected, as they all utilize RSBs to predict return addresses.

More on SpectreRSB:


The NetSpectre vulnerability is related to Spectre variant 1 in that it uses speculative execution to perform bounds-check bypass. NetSpectre can be used to defeat address-space layout randomization on a remote system.

Unlike other Spectre variants, NetSpectre does not require a victim to download and run malicious code their machine. Instead NetSpectre only requires the victim to have an active network connection. NetSpectre variations are currently too slow to be valuable to attackers, with exfiltration speeds between 15 and 60 bits/hr. Even if the attack vector is not currently usable, it demonstrates that Spectre is not simply a local vulnerability.

It’s reported that all CPUs affected by the Spectre variant 1 are also affected by NetSpectre, therefore existing vendor mitigations should be enough to protect against it. These mitigations require firmware and OS updates, so unpatched devices remain vulnerable.

More on NetSpectre:

New Freestanding C++ Proposal

In the June 2018 Newsletter, we mentioned the C++ standards proposal to support a "Freestanding" C++ language subset which is suitable for use without an underlying operating system.

A new version of the Freestanding proposal has been released: P1105R0: Leaving no room for a lower-level language: A C++ Subset.

The primary goal of the proposal is to make core language features (such as exceptions, RTTI, and thread local storage) optional if they have an OS dependency or incur storage space overhead. This would better-enable C++ to support the embedded systems and kernel programming communities.

For more on Freestanding C++:

Embedded Artistry's Technology Radar

I was introduced to the Technology Radar concept while at Bredemeyer Consulting's Software Architecture Workshop. The Technology Radar concept was created by ThoughtWorks, and they regularly publish updates to their own radar.

I immediately latched onto the concept as a way to keep track of the technologies and techniques that we are investigating at Embedded Artistry. The technology radar also helps us document the solutions that we recommend others adopt.

We’ve identified four categories that we plan to explore as an embedded systems consulting firm:

  1. Tools
  2. Libraries and Frameworks
  3. Platforms (both hardware and RTOS)
  4. Techniques

We've released the first-draft of our technology radar and will update it on an ongoing basis.

You can find our Technology Radar on the website under the Resources menu.

For more on technology radars:

Around the Web

IT Hare has published another article on (Re)Actors: CAS (Re)Actor for Non-Blocking Multithreaded Primitives.

An amusing RISC-V smear campaign by ARM has brought a lot of public attention to the RISC-V architecture. If you're unfamiliar with RISC-V, check out this IEEE article: RISC-V's Open-Source Architecture Shakes Up Chip Design. You can learn more on the RISC-V Foundation website.

Segger, who produces the J-Link and J-Trace tools, has announced their own C-based embedded OS called emPack. In addition to an operating system, Segger is also providing emPack plugins for GUI, File System, TCP/IP, USB-Device, USB-Host, and more.

Website Updates

We've launched a Store page, where we will be adding templates and informational products that embedded teams can leverage. Both free downloads and paid downloads are in the Store.

We've created a Support page which will stay updated with the ways you can support the website.

We updated the following blog posts this month:

Our Glossary is always expanding and has been updated with a plethora of new terms.

As mentioned we added a link to the Embedded Artistry Technology Radar under the Resources menu.

New Articles

These articles were published on our website in July:

These were our most popular articles in July:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. Jenkins: Running Steps as sudo
  6. An Overview of C++ STL Containers
  7. Implementing an Asynchronous Dispatch Queue
  8. A Simple Consulting Services Agreement
  9. Creating and Enforcing a Code Formatting Standard with clang-format
  10. Demystifying ARM Floating-Point Compiler Options

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter.

Happy hacking!


July 2018: Anniversary Edition

Welcome to the July 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month marks the one-year anniversary of our newsletter! In this edition, we’ll cover:

  • Looking back at major developments over the past twelve months
  • A new Intel vulnerability announcement
  • C++ conference talks with an embedded focus
  • The new I3C specification
  • Maxim's new DARWIN MCU line
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Looking Back

We’ve been publishing our newsletter for one year. The past twelve months have been eventful for embedded systems. Here are some of the events that stood out to us:

Intel Vulnerability Announcement

A new Intel vulnerability has been announced, dubbed "lazy floating-point (FP) state restore". This vulnerability affects all Intel Core-based microprocessors.

Intel provides a lazy floating-point state restore technique, which delays restoration of a processing state until an instruction operating on that state is executed by a new process. An attacker can take advantage of this technique to infer register values of a different process through a speculative execution side channel. This can be used to retrieve AES encryption keys, which are often stored in the floating-point unit (FPU) registers.

Intel recommends that developers use Eager FP state restore to prevent this vulnerability.

For more on lazy FP state restore:

Embedded C++ Talks

A lot of great Embedded C++ conference talks have been published over the past few months. Here are some of our favorites:

The I3C Specification

A typo led me to discover that the MIPI Alliance has announced the I3C Specification, the successor to I2C. I3C also intends to replace the SPI bus, which has no clearly defined standard and sports multiple implementations. Since GPIO availability is always contentious, many board designers will gladly switch to I3C's two-wire bus if it lives up to the hype. SPI will still maintain an advantage over I3C in high-speed point-to-point use cases, like SPI flash devices.

I3C is still a two-wire interface but unlike I2C, I3C will provide a minimum data rate of 10Mbps. I3C will also support multi-master, slave-initiated communication, dynamic addressing, and power management features (eg. sleep mode). A broadcast address is provided, allowing masters to send command codes to all slave devices at once. I3C lines will be push-pull instead of open-drain, enabling better power consumption per transferred bit. Most I2C slave devices will be able to operate on an I3C bus as long as they have a 50ns glitch filter and do not stall the clock.

I3C is brand new, so I have yet to find any sensors or processors that have shipped with I3C support. I expect we’ll start to see some devices announced later this year and more announced next year.

For more on I3C:

Maxim's DARWIN MCUs Line

Maxim has historically been in the power-IC business, focusing on components such as regulators, fuel gauges, and PMICs. They recently announced a low-power processing line: DARWIN.

The DARWIN MCU line is split into two families:

  • Generation U: Ultra-low power, with flash sizes from 256KB to 2MB, and SRAM sizes ranging from 96KB to 512KB
  • Generation UP: Ultra-low-power + performance, with 3MB flash and 1MB SRAM

The Generation U MCUs feature a Cortex-M4 with an FPU and can be clocked up to 96MHz. Maxim is capitalizing on their power management specialty with their MCUs. The processors include an embedded PMU, provide flexible power modes, and support dynamic clocking (down to 4MHz on the processor clock).

The Generation UP MCU is also a Cortex-M4. The UP MCU can be clocked up to 120MHz and includes SPI execute-in-place (XIP) interfaces. A memory decryption integrity unit (MDIU) provides on-the-fly decryption of data stored in external flash. Power management capabilities are further extended by providing power control over individual SRAM banks.

The Darwin MCU line looks to be an interesting contender in the low-power microcontroller space.

For more on the DARWIN MCUs:

Around the Web

ITHare published A Useable C++ Dialect That is Safe Against Memory Corruption. This article covers using (Re)Actors to help ensure memory safety in our programs.

All About Circuits takes a look at IP tension in Who Really Owns Hardware: Property Rights vs Copyrights

Another great All About Circuits article was Do Engineers Design Anything Anymore? Electrical Engineering in the Age of ICs.

Atomera plans to breathe new life into older chip manufacturing with a technique that involves burying atom-thin layers of oxygen below the surface of a transistor's silicon. This technique reduces variability among transistors and enables designers to make use of larger process technologies.

The Wi-Fi Alliance has announced the Enhanced Open certification to provide improved privacy protections and security on open Wi-Fi networks.

Website Updates

We have done some major revamping of our website this month!

New Articles

The following articles were published on our website in June:

These were our most popular articles in June:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. An Overview of C++ STL Containers
  5. std::string vs C-strings
  6. Jenkins: Running Steps as sudo
  7. A Simple Consulting Services Agreement
  8. Implementing an Asynchronous Dispatch Queue
  9. C++ Casting, or: Oh No, They Broke Malloc!"
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!


June 2018: C++ Standard Proposals for Rapperswil Meeting

Welcome to the June 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover 5 C++ language proposals that will impact embedded developers.

C++ Standardization Meeting in Rapperswil

Today marks the start of the C++ standardization committee meeting in Rapperswil, Switzerland. Developers from around the world will meet and work on the future of the C++ language.

C++ is complex, widely adopted, and must meet the needs of developers of all varieties. There are always competing priorities and needs, and it is up to the standardization committee (and an array of special interest groups) to ensure the proper balance is met.

I want to share five C++ proposals which are of interest to embedded systems developers. It may be years before we see these proposals in a formal C++ standard (if they are adopted at all), and the proposals should be expected to evolve with feedback from the standardization committee.

Even though they may be years out, these ideas can still benefit you today. Many of the proposals have corresponding testing libraries enabling you to try out the new features in your own programs. It's also exciting to take a look under the hood and see what future language changes are being discussed.

The proposals we will review are:

  1. P0709R0: Zero-overhead deterministic exceptions
  2. P0829R2: Freestanding
  3. P1028R0: status_code and standard error object
  4. P0037R5: Fixed-point real numbers
  5. P0132R1: Non-throwing container operations

For more on the Rapperswil meeting:

P0709R0: Zero-overhead Deterministic Exceptions

The Zero-overhead Deterministic Exceptions proposal is a potential game-changer for the C++ language.

The opening of Herb Sutter's proposal states that over 50% of C++ developers report that exceptions are banned in part or all of their code. As an embedded developer, I fall into both the partial- and complete-ban categories on my systems. Since exceptions are required to use many core C++ language features (e.g. constructors), this means that a large number of C++ programs are written in non-standard C++ dialects.

This proposal aims for a re-conception of the C++ exception model with two primary goals:

  1. Remove all technical reasons for a C++ project to disable exception handling
  2. Facilitate re-unification of divergent error reporting styles

The primary method of achieving the two goals is to annotate functions with the throws keyword, indicating that the function throws a statically specified type by value. Exception handling implementations can then be as efficient and deterministic as a local return-by-value, incurring no dynamic or non-local memory overhead.

How can this be accomplished? By ensuring that the only type of exception that can be returned is a std::error exception (discussed below in P1028). If the function fails, it will throw the std::error object as-if returning an alternate return value on the stack. Two additional possibilities are returning errors in unused processor registers and repurposing unused CPU flag bits to indicate whether the registers contain a return value or error.

The new exception mechanism provides two return paths for a function: one for the success case, and one for the failure case, where failure returns an error code as usual. The resulting API designs will be greatly simplified: no longer will error codes monopolize our return value channels, and users can't forget to check errors.

Those interested in trying out the new exception model are in luck. Niall Douglas has published the Outcome 2.0 library, which is effectively a library-only implementation of Herb's exception proposal.

For more on Zero-Overhead Deterministic Exceptions:

P0829R2: Freestanding

The Freestanding C++ proposal is one of the most interesting for the embedded world. Ben Craig is interested in identifying a subset of the C++ language which is suitable for kernel and embedded development.

The core goal is to enable developers to easily identify features which can be implemented without an OS. This is currently impossible with the existing standard, and many teams simply maintain a manual list of features which cannot be used in an embedded environment.

The paper identifies a working list of functions from existing headers (such as <cerrno>, <cstring>, and <algorithm>) which are allowed in a freestanding environment. The paper also identifies new headers which will need to be added and clearly outlines featuers which are not being considered for the freestanding subset.

Even if a Freestanding C++ is never formalized, the paper serves as a starting-point for features that are safe to use in an embedded environment. I will update the Embedded Artistry libc to incorporate some of the freestanding proposal notes.

For more on Freestanding C++:

P1028R0: status_code and standard error object

The standard error object proposal is a key component of Zero-Overhead Deterministic Exceptions. Even better, this proposal is compatible with the Freestanding C++ proposal, so all of the features can be used standalone on embedded systems.

This proposal covers the replacement of the <system_error> header with a lighter-weight and portable design. We also get a new std::error type with the following properties:

  • Always indicates a failure (no more "0" success codes)
  • No default construction
  • No empty state possible
  • Trivially copiable
  • Small enough to return in CPU registers (current proposal uses two pointers)
  • Portable across domains & operating systems

All operations which can be constexpr have been updated to be so, and the rest have been made literal or trivial to the maximum extent possible. Header-only libraries will now be able to safely define custom error code categories (you'll be unpleasantly surprised if you try this today). We can finally drop the if(!error_code) model, opting instead for a clearer if(ec.success()) or if(ec.failure()) check.

If you are interested in trying out the proposed status_code and std::error changes, you can try out Niall's status_code library. The Outcome 2.0 library, which demonstrates the exception proposal, also integrates the status_code support.

For more on the error code overhaul:

P0037R5: Fixed-Point Real Numbers

Fixed-point numbers are common in embedded systems for a variety of reasons:

  • Some systems don't have floating point support
  • Floating point support is often not thread-safe unless properly protected
  • Many external components communicate using a fixed-point format
  • The system may need to convert a number to fixed-point before sending the value out over a communication bus

Because there is no standard option available, developers are required to spin their own implementation or import one of the many fixed-point mathematics libraries.

Rather than treating fixed-point numbers as a special case, this P0037 proposes a drop-in integer replacement with fixed-point support. After all, an int is simply a fixed-point number with an exponent of 0. An API with the necessary fixed-point arithmetical tools will be provided by the standard.

John McFarlane's Compositional Numeric Library, which serves as a proof-of-concept for both P0037 and P0554, contains an in-development implementation of the proposed fixed_point class template. If you need fixed-point operations in your next project, give it a try.

For more on Fixed-Point Real Numbers:

P0132R1: Non-throwing container operations

As we covered earlier, many C++ projects operate with exceptions disabled. This is a problem since many of the standard library features are built around exceptions, especially allocators and constructors. If exceptions are disabled and an allocation fails, your program will typically end up in the abort function, which prevents any chance of error recovery.

P0132 proposes to create container operations which will not throw an exception on failure. The proposed additions to the standard are:

  • Allow an allocator to fail without an exception
    • Add a separate function for non-throwing allocation which returns nullptr on failure
  • Add a non-throwing reserve operation to vector and string
    • This would utilize the support for allowing an allocator to fail without an exception
  • Add a non-failing push_back to vector and string
    • Pushing without sufficient capacity will be undefined behavior
  • Add a push_back that can fail to vector and string
    • This function will report an allocation error, but will do so without an exception
  • Add a push_back that can fail to vector and string
    • This function will have no effects if allocation fails, and will not throw

As an embedded C++ developer, I would love to see these features and look forward to improved container operations in the coming years. I haven't yet seen a library to test such support, which means you will need to modify your local libc++ implementation to try these features out.

For more on Non-throwing Container Operations:

Around the Web

Watch John Regehr's closing keynote from C++Now 2018: Undefined Behavior and Compiler Optimizations.

GitNStats is a tool for analyzing how often particular files are modified. By identifying files with a lot of churn, you identify potential problem areas in your code base.

Jacob Benigo postulates on the soon-to-be-extinct embedded software engineer.

Sensirion recently announced the SDP3x differential pressure sensor family, which they claim is the world's smallest differential flow sensor.

Microsoft has announced Vcpkg, a C++ library manager for Linux, macOS, and Windows.

Website Updates

The For Beginners page has been reorganized, and a table of contents has been added.

The Hardware References page now has a table of contents. I also added Circuit Cellar's compendium of quick prototyping solutions.

The Software References page has been reorganized and a table of contents has been added. Paul Watt's blog was added to the software blog list.

New Articles

These articles were published on our website in May:

These were our most popular articles in May:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. Jenkins: Configuring a Linux Slave Node
  4. std::string vs C-strings
  5. Jenkins: Running Steps as sudo
  6. A Simple Consulting Services Agreement
  7. An Overview of C++ STL Containers
  8. C++ Casting, or: Oh No, They Broke Malloc!"
  9. Implementing an Asynchronous Dispatch Queue
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!


May 2018: C++ Embedded Frameworks

Welcome to the May 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Four open-source C++ embedded frameworks that can jump-start your next project:
    • Kvasir
    • modm
    • crect
    • EMB2
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Open-Source Embedded Frameworks

In last month's newsletter we recommended quality C and C++ libraries to give your next project a boost.

This month we are taking things a step further by diving into embedded C++ frameworks which can be used as the foundation for your next product or project. While each framework is focused on solving a distinct problem, there is at least one that will benefit your team.

We will cover the following four embedded frameworks:

  1. Kvasir, which specializes in compile-time checking of Special Function Register usage
  2. modm, which utilizes vendor-provided data to generate drivers and BSPs
  3. crect, which generates a deadlock-free and data-race-free scheduler for real-time systems at compile time
  4. EMB2, which provides a paradigm for developing multi-core and mutli-chip embedded programs


Kvasir is a C++11 framework created by Odin Holmes. Kvasir is suitable for tiny bare-metal applications such as ARM cores. By using C++11’s constexpr feature and template meta-programming, Kvasir’s features come at no cost. In some cases, the meta-programming magic provides superior implementations to common hand-coded approaches!

The primary feature is the Kvasir::Register abstraction, which enables strongly typed register and bit-field definitions and provides SFR read/write methods. These abstractions improve type safety and enable static checking of register interactions during compilation. As long as the register definitions are correct, Kvasir will eliminate off-by-one errors, prevent users from writing to reserved bits, ensure that bit-field accesses don't overflow into other fields, and generate warnings if you are incorrectly using a register that performs a clear-on-read operation.

In addition to SFR abstractions, Kvasir also supplies the Kvasir::IO abstraction to provide similar benefits with GPIO. The IO abstraction utilizes a pinLocation concept to define GPIOs and provides constexpr meta functions such as makeOpenDrain(), makeOutput(), set(), clear(), and read().

SFR bugs are extremely common on embedded systems. Eliminate them altogether by building your next system with Kvasir.

For more on Kvasir:


modm (Modular Object-oriented Development for Microcontrollers) is a C++14 framework built by Niklas Hauser and Fabian Grief. The modm project uses vendor-provided chip data with a library builder, enabling modm to automatically generate startup code, chip-specific drivers, external drivers, and BSPs for your platform. Since modm provides a portable HAL, you can easily migrate your software from one processor to another supported processor with minimal effort.

modm provides a framework which is suitable for bare-metal systems ranging from the ATtiny to a 32-bit ARM Cortex-M. The HAL features no memory allocations, low RAM consumption, and lightweight stack-less threads. The framework also provides useful algorithms suitable for bare-metal systems, as well as drivers for a wide variety of SPI and I2C peripherals. modm is well-tested, featuring 64 test groups with 343 test cases and over 4000 assertions. While the HAL is not fully tested in an automated manner, a variety of example hardware projects are regularly checked by the CI server.

The modm framework currently supports ~1350 AVR and ARM Cortex-M microcontrollers from Atmel, ST, and NXP. If you are using a processor from of those vendors, modm can provide your team with a stable foundation of drivers and the advantage of being able to quickly migrate your software to another processor.

For more on modm:


The crect project (pronounced correct) is a C++14 framework for generating a scheduler for Cortex-M microcontrollers at compile-time. crect uses the Cortex-M's Nested Vector Interrupt Controller (NVIC) to implement a Stack Resource Policy (SRP) scheduler which guarantees deadlock-free and data-race-free execution.

crect is built upon the Kvasir Meta-programming Library, which is also the foundation of the Kvasir framework. The use of C++ meta-programming and C++14 features allows priority ceilings and interrupt masks to be calculated at compile time. Resource locks are handled through RAII, and resource access is handled using a monitor pattern).

The most impressive thing about this framework is the minimal resource requirements:

  • ~400 bytes of memory for the static elements (linked list, SysTick, time implementation)
  • 4-5 instructions per job for initializing the NVIC
  • 2-3 instructions per queue element for initializing the asynchronous queue
  • 3-4 instructions + 4 bytes of stack space for a lock
  • 1-3 instructions for an unlock
  • 2-4 instructions for a pend/clear
  • 20-30 instructions per item in the queue for async

If you are working on a bare-metal ARM program with real-time concerns, crect is an RTOS alternative that provides protection against common multithreading concerns like priority inversion, deadlocks, and race conditions.

For more on crect:


We discussed the EMB2 project in the November 2017 newsletter in the context of Industry Standard Multicore APIs. I am including EMB2 again here to emphasize its importance as a modern embedded systems framework.

EMB2 is a framework developed by Siemens and the University of Houston. EMB2 provides generic building blocks for building multi-core or multi-chip embedded applications, including basic parallel algorithms, concurrent data structures, and application skeletons. Since EMB2 is targeted for embedded applications, it provides soft-real-time support, predictable memory consumption (no dynamic memory allocations after startup), non-blocking APIs, and support for task priorities and affinities.

The framework utilizes the MTAPI abstraction layer, enabling EMB2 programs to be easily ported to new operating systems and processor architectures. Heterogeneous and distributed embedded programming is simplified by utilizing MTAPI. Developers can easily distribute work across processor cores, hardware accelerators, GPUs, DSPs, FPGAs, or networked devices.

The EMB2 base library is implemented as a C API with C++ wrappers. While the parallel algorithms, dataflow patterns, and concurrent containers are implemented in C++. C99 and C++03 are used as the implementation standard to provide maximum usability in the embedded world. C11 and C++11 are also supported.

If you are building a product which uses a multi-core processor, multiple processors, or hardware accelerators, EMB2 provides a solid and portable foundation that will enable your team to take full advantage of your system's hardware resources.

For more on EMB2:

Around the Web

The Industrial Internet Consortium published their IoT Security Maturity Model. If your team is working on an IoT device, this model will help you figure out where you need to be on the security spectrum and how to properly invest in the relevant security requirements.

Thinking about cutting corners or ignoring security in your IoT product? You might be featured in the IoT Hall of Shame.

Niklas Hauser, the author of modm, has two great articles for improving your embedded C++ skills:

Website Updates

The Archive now has a Guest Posts section so our readers can find our external publications.

Additional terms have been added to the Glossary: SRP, SFR, Cortex-A, Cortex-M, RTFM, NVIC

The Software References page was expanded with a variety of embedded and software development links.

Hardware rules of thumb were added to the Hardware References page.

The For Beginners page was expanded with C pointer references, code review articles, and introductory GitHub resources.

New Articles

These articles were published on our website in April:

I also shared some of my personal business experience with the Harpoon team in a Customer Spotlight.

These were our most popular articles in April:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. Jenkins: Configuring a Linux Slave Node
  4. C++ Casting, or: Oh No, They Broke Malloc!"
  5. std::string vs C-strings
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Demystifying ARM Floating Point Compiler Options
  9. Implementing Malloc: First-fit Free List
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!


April 2018: C & C++ Libraries We Like

Welcome to the April 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Seven exceptional C and C++ projects to consider for your next product
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Open Source Projects to Consider For Your Next Product

I’ve selected seven exceptional C & C++ projects that I want to share with you. Integrating high-quality libraries in our code base enables us to take new approaches, use safer techniques, or simply reduces the amount of code we need to write. I hate reinventing the wheel unless there’s a compelling reason, so I like to spend time researching libraries before diving into development. I hope you can find something to benefit your next product.

C++ Projects

I primarily focus on developing new embedded systems in C++. The language provides features that allow me to write safer code and to detect errors in compilation rather than at runtime. Of the seven project we’ll review today, 5 of them are C++:

  • NamedType
  • foonathan/type_safe
  • foonathan/memory
  • POCO C++ Libraries
  • Safer C++


NamedType is a library written by Jonathan Boccara, a C++ developer and author of the FluentC++ blog. The NamedType library provides a simple interface for using strong types, which are not natively supported in C++.

We often utilize native types, such as int or double, to represent the values in our software. The general nature of these types means that we can easily make mistakes in our code. Consider a simple rectangle constructor:

Rectangle(double width, double height);

If you swapped the width or height in your code, the compiler would never be able to tell you.

double width = 10.5;
double height = 3.0;
Rectangle r(height, width); // wrong but compiles!

By using strong types in our interfaces, we can make our APIs more explicit and rely on the compiler to catch any mistakes:

// Create our strong types
using Width = NamedType<double, struct WidthTag>;
using Height = NamedType<double, struct HeightTag>;
Rectangle(Width width, Height height); // new constructor
Rectangle r(Height(3.0), Width(10.5)); // compiler error - type mismatch!
Rectangle r2(Width(10.5), Height(3.0)); // ok

The great news is that strong types are zero-cost when compiling with -O2 (or -O1 with GCC). It's time to make your APIs more explicit and catch errors during compilation.

For more on NamedType and strong types in C++:


The type_safe library is developed by Jonathan Müller, a C++ library and author of foonathan::blog(). The type_safe library provides zero-overhead utilities to help catch bugs at compile time.

Features provided by the type_safe library include improved built-in types (ts::integer, ts::floating_point, and ts::boolean) which prevent dangerous implicit operations like signed-to-unsigned promotion. The type-safe library also provides "vocabulary" types to help us write more expressive code, such as object_ref (non-null ptr), array_ref (reference to an array), and function_ref (reference to a function). Other interesting concepts are also provided, such as deferred_construction, a wrapper which allows you to create an object without constructing.

Similar to NamedType, this library also supports strong types. Where NamedType provides a simple interface for creating strong types, the type_safe library requires explicit declaration of the attributes our strong type supports. The increased overhead for setting up new types is worth the safety provided by explicitly deciding what operations should be allowed. In the example below, our strong type only has addition and subtraction operations enabled:

struct meter
: strong_typedef<meter, int>, addition<meter>, subtraction<meter>
    using strong_typedef::strong_typedef;

Use the type_safe library to write expressive code and increase the number of errors caught during compilation.

For more on type_safe:


The memory library is also developed by Jonathan Müller. This library provides an new STL-compatible C+ memory allocator called RawAllocator. The RawAllocator is similar to the standard Allocator but is easier to use. The library also provides a BlockAllocator type which can be used for allocating large blocks of memory.

The project includes a variety of implementations, adapters, wrappers, and storage classes, including:

  • new allocator
  • heap allocator
  • malloc allocator
  • memory pools
  • static allocator
  • virtual memory allocator
  • make_unique and make_shared replacements which allocate memory using a RawAllocator

We are excited about using this library in our next embedded project and gaining increased control over memory allocations.

For more on the memory library:

POCO C++ Libraries

The POrtable COmponents (POCO) C++ Libraries is a collection of C++ class libraries whose goal is to simplify the development of network-centric applications. POCO is similar in concept to the Java Class Library, .NET Framework, or Cocoa. The POCO libraries are highly portable, allowing you to easily compile and run your application on multiple platforms.

The full list of features provided by the POCO libraries are too numerous to list here. A small sampling of features:

  • Caching framework
  • Cryptographic & hashing libraries
  • Logging framework
  • HTTP server & client
  • SSL/TLS support through OpenSSL
  • POP3 & SMTP clients
  • SQL database access
  • Multithreading (basic threads, synchronization, thread pools, active objects, work queues)
  • Stream classes for Base64 and binary encoding/decoding, compression (zlib), line ending conversion
  • XML parsing & generation
  • Zip file manipulation

While the POCO libraries provide portability and a plethora of features, my favorite aspect is the emphasis on code quality, style, consistency, and readability.

I highly recommend reviewing the POCO libraries before beginning your next project - you can find a gem (or twenty) that will save you development time.

For more on POCO:


The SaferC++ library provides safer implementations for many native C++ types. The library provides features such as:

  • Data types that are designed for multi-threaded use and asynchronous access
  • Drop-in replacements for std::vector, std::array, std::string, std::string_view that provide improved memory safety
  • Drop-in replacements for int, size_t, and bool that protect against use of uninitialized values and sign-unsigned comparison issues (similar to type_safe)
  • Improved pointer & reference types with different compatibility and performance tradeoffs

SaferC++ is usable with embedded systems as long as your platform has a functional STL implementation. Exception behavior can be controlled for your platform by modifying the MSE_CUSTOM_THROW_DEFINITION macro.

Using the library does incur a performance penalty. However, SaferC++ elements can be disabled during compile time (i.e. replaced with the standard type equivalents). This allows users to enable debug and test builds to use safer-but-slower features without adding overhead to release builds.

Since the SaferC++ types provide added safety and can be disabled when performance matters, I highly recommend using their drop-in types to catch and eliminate possible errors when using STL types. The easiest way to get started with SaferC++ is to utilize the mse::vector and mse::array types in place of std::vector and std::array. These types will help you catch potential memory issues lurking in your software. The README provides further tips for making your code safer.

For more on SaferC++:

C Projects

While we’ve been heavily focused on C++-based embedded systems development for the past few years, I did find two exciting C projects that I want to share with you:

  1. CException, a lightweight exception library
  2. Checked C, an extension to the C language which provides better protections against common memory errors


CException is a project released by Throw The Switch. CException is designed to provide simple exception handling in C using the familiar try/catch/throw syntax. I've been recently thinking about the downsides of error logging, so I am excited to see a lightweight exception library for C.

The exception implementation is kept simple by only allowing you to throw a single error code. No support for throwing objects, structs, or strings is included. The library can be configured for either single-tasking or multi-tasking which makes this project a good fit for embedded systems using an RTOS. CException is implemented in ANSI C and is highly portable. As long as your system supports the standard library calls setjmp and longjmp, you can use CException in your project. If you're looking for an exception library to use on embedded systems, CException is for you.

For more on CException:

Checked C

Checked C is a research project from Microsoft which adds static and dynamic (runtime) checking for errors such as buffer overruns, out-of-bounds memory accesses, and incorrect type casts.

The project is implemented as an extension to the C language. New pointer and array types are provided with the goal of allowing programmers to better describe intended pointer use and the range of memory that is pointed to. Developers can select between types with and without bounds checking, as well as between types that can or cannot be used in pointer arithmetic.

Since Checked C is an extension to the C language, you will need a compiler that supports it. Microsoft provides a port of clang and LLVM that support the extension.

CheckedC can help you identify and eliminate common memory errors which plague us as C & C++ developers. Even better, existing C programs compiled with a Checked C compiler will continue to work. Raw pointers (e.g. int *) remain unchecked and pointer arithmetic is still allowed.

For more on Checked C:

Around the Web

In the September 2017 newsletter I shared a series of posts detailing the engineering behind the Voyager missions. If you liked those articles, Voyager Mission Telecommunication Firsts presents another take on the outstanding engineering achievements of the Voyager missions.

If you enjoyed last month's focus on development processes, I recommend reading Making Valgrind Easy. By integrating Valgrind into your static analysis process, you can find and fix memory issues that your compiler or static analyzer won't catch.

I came across this article written by Michael Barr in 2009, titled Firmware Architecture in Five Easy Steps. Read this article before starting your next embedded project.

Jonathan Müller of foonathan posted Guidelines for Rvalue References in APIs. This article is recommended for advanced C++ developers and library authors.

Website Updates

Our "About" information has been condensed into a single page. We've doubled down on keeping the primary website focused on embedded systems content. We've created a new website dedicated to our consulting business.

A classic (and free!) introductory embedded systems book, Programming Embedded Systems, was added to the Beginners page. While the examples are slightly dated, the concepts are valid and can be applied to modern embedded systems.

The Glossary saw new additions, including LVDS, PLM, and MIPI standards.

The old "Open Source Software" page has been merged into the Libraries page.

New Articles

The Dark Side of Error Logging was published as a guest post on Arne Mertz's Simplify C++ blog.

These posts were published on our website in March:

  1. Improving Our Software With 5 Lightweight Processes You Can Adopt This Month
  2. Getting Started with the Snapdragon Flight: Driver Development
  3. Seeing Intermittent GitHub Clone Failures on Jenkins? Check Your Repo Size
  4. Safely Storing Secrets in git

These were the most popular articles in March:

  1. Circular Buffers in C/C++
  2. C++ Casting, or: Oh No, They Broke Malloc!"
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. Jenkins: Configuring a Linux Slave Node
  6. An Overview of C++ STL Containers
  7. Implementing Malloc: First-fit Free List
  8. Demystifying ARM Floating Point Compiler Options
  9. Jenkins: Running Steps as sudo
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!


March 2018: Lightweight Processes to Improve Quality

Welcome to the March 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • 5 methods for improving software quality that you can implement this month
  • The keys to success when adopting new processes
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Improving Our Software With 5 Lightweight Processes You Can Adopt This Month

It's that time of year when the Barr Group releases their yearly Embedded Systems Safety & Security Survey results. Last year's results were eye opening, as nearly 50% of respondents reported not using static analysis and 36% reported that they do not perform code reviews. The 2018 results were no better:

  • 38% of safety-critical products don't comply with a formal safety standard
  • 43% of teams working on safety critical products don't perform code reviews
  • 41% of of teams working on safety critical devices don't perform regression testing (54% for IoT product teams)
  • 33% of teams working on safety-critical products don't perform static analysis (49% for IoT product teams)

These numbers are even more alarming given the fact that 25% of the reported "internet-connected devices" could kill or injure people if hacked. 22% of respondents mentioned that security for connected devices wasn't even on their to-do list. We're becoming increasingly connected, but our standards for safety, testing, and verification are not keeping pace. I want to be clear: this is not acceptable.

Many teams skip crucial development processes and justify them with scheduling pressure or by blaming the boss. There will always be scheduling pressure, so we must adjust our approaches or we will continue to flounder. Bugs are expensive in time, money, and morale. Debugging accounts for 50% of most project schedules. We all make mistakes. Anything we can do to keep bugs out of our code or catch them as early as possible will save money and time.

I've selected five simple processes to improve quality that your team can adopt over the next month. These processes are cheap or free, apply across languages and platforms, and best of all - they work. While each process requires a bit of time to get up and running, there is little-to-no maintenance involved in continuing to use them.

Here are five lightweight processes for improving code quality and identifying problems early:

  1. Fix all of your warnings
  2. Set up a static analysis tool for your project
  3. Measure and tackle complexity in your software
  4. Create automated code formatting rules
  5. Have your code reviewed

Fix All of Your Warnings

The first thing I do when working on a new project is fix all of the compiler warnings. It's amazing to me how developers will ignore warnings or rationalize their presence. Occasionally you even run into a team who will fight tooth and nail to prevent you from fixing them!

The compiler knows the programming language better than you ever will. You should not ignore the compiler when it is alerting you to an issue. The way you are using the language is dangerous and likely has unintentional side effects. Depending on the warning, you might be introducing undefined behavior into your software. That's not our idea of quality software.

If you have warnings in your code base, fixing them is one of the fastest ways to improve quality. You will fix bugs and flaws in your program, regardless of whether or not they are currently problematic.

For more on compiler warnings:

Set Up Static Analysis Support

Static analysis tools provide us with even better feedback than the compiler. Your compiler will happily allow some problematic cases which are legal in language, such as out-of-bounds pointer accesses or missing initialization values. Your analyzer will catch these problems, and also report red flags such as unused or redundant code. When used throughout the development cycle, your static analysis tool can help you catch & prevent latent problems even earlier than your testing cycle.

Some governmental and industrial organizations are starting to require static analysis data for certification processes. Companies such as PRQA provide tools that can check for compliance with safety critical standards in a variety of industries.

There are many free static analysis tools available and their commercial counterparts are also inexpensive (most are less than $1000). At Embedded Artistry, we use Clang's static analyzer alongside clang-tidy.

Here are some resources you can use to find a static analysis tool that fits your needs:

Measure and Tackle Complexity

By the time you've eliminated warnings on your project and cleaned up glaring problems exposed by your static analysis tool, you've already made significant progress with software quality. The next goal is to measure complexity in your software. Because highly complex functions tend to be hard to understand, test, and maintain, these functions are prime candidates for refactoring and simplification.

By using a metric to measure complexity, we have a quantitative way to evaluate our code and identify pieces that need special attention. We can see how our changes are impacting the code base over time and trigger automatic alerts and reviews whenever a threshold is exceeded. We can focus our code reviews on functions with high complexity scores, making sure they get the most of our limited attention. Metrics aren't perfect, but they increase our insight into our software quality.

These are the simplest and most popular metrics for measuring code complexity:

  • Lines of code (LOC): a count of the non-blank, non-comment source lines in a function, module, or project
  • McCabe cyclomatic complexity (MCC): provides a complexity score based on the number of branches (e.g. conditional statements)
  • Strict cyclomatic complexity (SCC, CC2): expands MCC by considering the number of conditions within each branch, which provides an approximation for the number of test cases needed for full coverage

These free tools will calculate complexity metrics for C/C++. We currently use Lizard at Embedded Artistry.

For more on software complexity:

Create Auto-formatting Rules

Automated code formatting might seem like a strange recommendation to put into the top five, but it serves three purposes:

  1. Automated formatting reduces a programmer's cognitive load by eliminating an entire category of details and decisions they need to keep in mind
  2. Automated formatting improves the quality of our peer code reviews (the next recommendation) by eliminating arguments about style
  3. Automated formatting is the first step toward implementing and enforcing a coding standard

Every team that I've encountered with a written style guide inevitably ignores those guidelines, and multiple programming styles run rampant. Instead of relying on developers to constantly keep an arbitrary set of rules in mind, we can automate the process to make it simple and impersonal. At the very least, it's worth eliminating the pointless, time-wasting arguments that cause friction within our teams.

We use clang-format on Embedded Artistry projects. Uncrustify and Astyle are other popular code formatting tools.

For more on automated code formatting:

Have Your Code Reviewed

Writers accept the fact that first drafts are generally garbage and need heavy editing. Before I send this newsletter out into the world, it has usually gone through 2-3 self-editing sessions and 1-2 peer reviews. Along the way, the newsletter is trimmed and restructured. The result is a much better product than the initial draft.

Yet, for some reason, programmers seem to think that perfect code is produced on the first try. The 2018 Barr Group survey results showed that 54% of IoT product teams don't perform regular code reviews. The survey results also show that a painfully scary 43% of teams working on safety-critical software don't perform regular reviews.

Perfect code on the first try might be possible if you're a prodigious programmer. But remember: even Hemingway had an editor. A second set of eyes can identify flaws that you missed in your first pass. Another developer may have different experiences that provide insight into the merits or risks of your approach. The architect on your team probably has input on how a module should interface with other pieces of the program. The "ego effect" also comes into play: knowing that our code will be presented and reviewed by another human can dramatically improve the overall quality. We will spend time cleaning up and checking the logic before putting code up for judgment.

Code reviews can waste time and become unproductive if poorly implemented. Best Practices for Peer Code Review provides some excellent tips for getting started. Notably, a lightweight review process is more efficient and practical than long, in-depth reviews with multiple developers. Even performing reviews on only 20-33% of the submissions provides benefits due to the "ego effect". While 20% may seem low, remember that we are aiming for achievable: reviewing 20% of the source code is definitely better than none.

I highly recommend implementing peer code reviews after setting up automated code-formatting. This helps constrain code review discussions by preventing them from devolving into style nit-picking. If you've set up static analysis, make sure the tools are used prior to code reviews.

For more information on code reviews:

The Keys to Success When Adopting a New Processes

When adopting new processes, it's important to focus on one at a time. Adopting new processes in stages ensures that you have time to correctly implement each new technique before moving on to the next one. By implementing too many changes at once, you are likely to overwhelm your team and evoke a mutiny.

If you're leading a team, it helps to find someone who is excited and can help you champion the idea. Empower that person so they can demonstrate the benefits of the new process to your team. Back them up when there is pushback. Change is always hard. Expect the resistance, but don't let it stop you.

The key to making new processes stick is to make them as automated as possible. There is never a case where it isn't worth the time it takes to automate a development process. Automation ensures that the process is easy to follow and always happens, rather than trusting individual contributors to remember to follow a process. Automation also makes the process less personal. The rules are clearly defined and are being enforced by a tool. Depersonalization helps us view the situation dispassionately, rather than as an attack on our abilities.

To recap, when you implement new processes:

  1. Adopt one new process at a time
  2. Empower a process champion on your team
  3. Automate!

Around the Web

BMW has issued a recall for 11,700 engines because they flashed the wrong firmware onto them.

In 2007, doctors replacing Dick Cheney's heart defibrillator ordered the manufacturer to disable its wireless capabilities out of concern for a hacker being able to trigger a fatal heart shock. We need to take the security of our connected devices, especially medical ones, seriously.

IT Hare published an extremely interesting analysis of operation costs in CPU clock cycles. Their data provides a great source of ballpark numbers for the relative computational costs of popular operations.

Michael Barr shared the 2018 survey results for programming languages used to develop embedded systems. C remains the main player with C++ in a distant second.

Website Updates

We've added two great introductory embedded systems resources by to the For Beginners page. If you're just starting out with embedded systems, check out Embedded Wednesdays and Embedded Software Engineering 101.

The Glossary has been expanded with additional embedded systems terms.

The Software References page has been updated with a TLA+ reference for those interested in verifying that their algorithms work correctly.

Articles Published in February

These posts were added to the website in February:

  1. Implementing std::mutex with ThreadX
  2. Implementing std::mutex with FreeRTOS
  3. Refactoring the ThreadX Dispatch Queue to Use std::mutex
  4. Code Cleanup: Splitting Up git Commits in the Middle of a Branch
  5. Generating GStreamer Pipeline Graphs

These were the most popular articles in February:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. C++ Casting, or: Oh No, They Broke Malloc!"
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Implementing std::mutex With ThreadX
  7. Implementing Malloc: First-fit Free List
  8. Creating and Enforcing a Code Formatting Standard with clang-format
  9. Implementing an Asynchronous Dispatch Queue
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!


February 2018: Spectre and Meltdown

Welcome to the February 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Two vulnerabilities that have rocked the computing world: Spectre and Meltdown
    • An overview of speculative execution
    • The Spectre vulnerability
    • The Meltdown vulnerability
    • What you can do today
    • Reliable sources for more information
  • Interesting links from around the web
  • Articles Published in January
  • Most Popular articles in January

A Tale of Two Vulnerabilities: Spectre and Meltdown

The new year did not get off to a great start for the electronics industry. A design flaw in modern processor architectures was exposed with the announcement of two critical vulnerabilities: Spectre and Meltdown. Both attacks are based on speculative execution, a technique in modern process design used to increase performance by pre-loading memory and future CPU instructions. Processor design has been largely focused on improving performance, and unfortunately the security implications of these improvements were not questioned. Spectre and Meltdown show that attackers can exploit speculative execution to access arbitrary memory locations.

Collectively, Spectre and Meltdown affect most processors that are on the market today. While Meltdown is a potentially patchable issue affecting most Intel and some ARM processors, Spectre can only be fully resolved by re-thinking our processor designs.

An Overview of Speculative Execution

Modern processors utilize instruction pipelines, effectively breaking up incoming instructions into sequential stages to keep the processor continually busy. Each pipeline stage handles its part of the current instruction, passes the result to the next stage, then handles the next instruction. This means that a processor effectively is working on multiple instructions in parallel.

The problem with instruction pipelines arises when a conditional branching decision is encountered. Consider the following simple statement:

if(x < 10)

The condition if(x < 10) must be evaluated before the processor knows whether it needs to call function A() or function B(). The pipeline stalls until the correct path is chosen, and valuable computational cycles are wasted.

Speculative execution is an optimization technique that tries to prevent those wasted computational cycles. Instead of stalling, the processor executes beyond the branch point (e.g. by calling A()) before it knows whether the branch will be taken. If the speculation was correct, then we gained the advantage of not wasting any computational cycles. If it was incorrect, the CPU discards the resulting state and continues executing on the correct path.

Note that the "speculated state" is not discarded until the correct execution path is known. This opens up the mis-speculation window, the time in which the CPU has speculatively executed the wrong code while not detecting that a mis-speculation has occurred.

Compounding with the mis-speculation window is caching, another performance optimization. When data is loaded into memory, caches are updated. If we need to refer to that memory again in the future, we can reduce the access time by fetching that information from the cache instead.

However, once the memory state is rolled back after a mis-speculation is detected, the data for the speculated state is still present in the cache. Clever attackers can take advantage of these speculative execution breadcrumbs to execute code and access memory that is not normally accessible.

There are currently three known variants of speculative execution vulnerabilities:

  1. Bounds check bypass (Spectre)
  2. Branch target injection (Spectre)
  3. Rogue data cache load (Meltdown)

At a high level, the Spectre exploits can be used to trick processors into running instructions they should not have run, granting access to arbitrary memory from another program's memory space. Meltdown applies primarily to Intel processors and can be used to access protected kernel memory in user space.

The Spectre Vulnerability

Spectre's name reflects the vulnerability's root cause (speculative execution) as well as the fact that it will haunt us for years to come. If your processor utilizes speculative execution techniques, you are probably affected. This includes the Intel, AMD, and ARM processors that power our personal computers, smart phones, and cloud servers. There are two primary variants of the Spectre vulnerability: branch target injection and bounds check bypass.

The branch target injection method relies on influencing how a processor's branch predictors operate. By influencing the branch predictors an attacker can control the speculative execution path and ensure that the malicious code is speculatively executed.

The bounds check bypass method takes advantage of speculative execution that occurs while the processor is checking if the targeted memory location is in-bounds. The processor will speculatively access out-of-bounds memory before the bounds check resolves. The attacker can read normally inaccessible memory by using a bounds check bypass combined with an intermediary program or module with better memory access privileges.

The end result is the same: processors inadvertently grant access to arbitrary memory from another program's memory space by running speculative instructions. After the code has been speculatively executed, an attacker can use the caches to reconstruct the target data.

The Meltdown Vulnerability

Meltdown is named for the fact that it "melts" normally enforced security boundaries between user memory and kernel (system) memory. Meltdown affects Intel, Qualcomm, and some ARM cores. Since Intel chips form the basis of many server platforms, most of the cloud service providers are affected by Meltdown.

Meltdown is only possible on processors which allow speculative execution across "privilege boundaries", such as the boundary which separates kernel memory from normal user programs. Normally, direct kernel memory access from user space is expected to fail with a page fault access error. However, certain processors might speculatively access the protected memory and use it for subsequent instructions prior to finishing the permissions check. If the permissions were not correct, a flag would be set and an exception thrown.

However, the kernel memory was still speculatively accessed and available in the cache during the mis-speculation window, allowing kernel memory to be read from user space.

What You Can Do Today

Your system is at risk if someone can run malicious code on your machine. This includes your personal computers, your phone, systems with multiple accounts, servers, cloud platforms, and virtualized environments. If your system is isolated from a network and only has a single user account, you probably don't need to worry about this exploit. If you're unsure of whether or not you need to take action, WindRiver's CEO provided a detailed framework for deciding how to respond.

If you are using an Intel chip or a cloud platform, make sure to stay up to date on patches for your system. The KPTI/KAISER patches for Meltdown have been applied to the latest Linux kernels but may need to be migrated to your specific kernel version. Similar patches have also been made for Windows and OS X. ARM has announced that some of its chips are affected by Meltdown, so check and see if your platform is affected.

Spectre is a trickier beast, especially since most modern processors are affected. A complete resolution is only possible through re-architecting the way processors are designed. We will be living with this threat for many years. Spectre-hardening software efforts are in progress, including ARM speculation barrier, MSVC mitigations, and LLVM mitigations.

Solutions for both of these vulnerabilities will be evolving over time, so be sure to stay up-to-date if you are affected.

Reliable References

The announcement has been surrounded with quite a bit of drama: CERT's initial recommended solution was to "replace CPU hardware", Linus Torvalds called Intel's patches "insane" and "complete garbage", and Intel admitted that its patches for Meltdown and Spectre are flawed.

Given all this drama, here are reliable sources you can refer to for more information:

Around the Web

For some non-Spectre-and-Meltdown news, check out the following articles.

Nordic Semiconductor has announced a new nRF91 chipset targeted for low-power IoT systems. The nRF91 System-inPackage (SiP) integrates an LTE cellular modem and transceiver, ARM Cortex-M33 processor, flash memory, RAM, and power management into a single package.

The Vancouver startup Riot Micro announced the RM1000, a low-power cellular baseband chip targeted for IoT applications.

Michael Barr proposes dropping the term "bug" in "Is it a Bug or an Error?". Jack Ganssle shared similar thoughts in "I've Never Had a Bug in My Code"

A new C++ programming language standard is now available.

Arne Mertz of Simplify C++ published a series on the topic of code reviews. The humble code review is one of the most powerful tools that teams can utilize to improve software quality and reduce errors. How can your team improve its review process?

  1. Code Reviews - Why?
  2. Code Reviews - What?
  3. Code Reviews - Preparation
  4. Code Reviews - How?
  5. Code Reviews - The Human Aspect

Articles Published in January

These posts were added to the website in January:

  1. Implementing an Asynchronous Dispatch Queue With FreeRTOS
  2. Getting Started with Snapdragon Flight: Dev Environment Setup & Useful Resources
  3. Implementing an Asynchronous Dispatch Queue with ThreadX
  4. Implementing malloc with FreeRTOS
  5. Jenkins: Configuring a Linux Slave Node
  6. Installing ROS on an NVIDIA Tegra TX2

Popular Articles

These were the most popular articles in January:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. C++ Casting, or: Oh No, They Broke Malloc!"
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Implementing Malloc: First-fit Free List
  7. Creating and Enforcing a Code Formatting Standard with clang-format
  8. A Simple Consulting Services Agreement
  9. A GitHub Pull Request Template for Your Projects
  10. Implementing an Asynchronous Dispatch Queue

Happy hacking!


January 2018: Component Counterfeiting

Happy New Year! Welcome to the January 2018 edition of the Embedded Artistry Newsletter. This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll be taking a look at the problem of electronic component counterfeiting and new anti-counterfeiting developments. We'll cover:

  • The problem of component counterfeiting
    • Detecting counterfeit components
    • New DARPA anti-counterfeiting initiatives
    • Using blockchains to establish a component "chain of trust"
    • Steps you can take to protect against counterfeit components
  • Recommended articles from around the web
  • Embedded Artistry website updates and popular posts

The Problem of Component Counterfeiting

Hardware designers face a variety of challenges today. Critical hardware components, such as NAND, DRAM, and OLED displays are experiencing shortages and long lead times. Companies are increasingly compressing schedules and striving to reduce the cost of producing their products. Driven by these schedule, price, and supply constraints, engineers and manufacturers will often acquire components from smaller distributors, electronic markets, scrap electronics dealers, or even eBay. The largest risk of using these untrustworthy sources for component purchases is the risk of receiving counterfeit parts.

Counterfeit components are introduced in a variety of ways, such as recycling old components from end-of-life products, recycling scrap electronic material, selling out-of-spec components, selling factory rejects, creating a cloned part, remarking parts with a higher-grade (e.g. commercial-grade parts marked as industrial-grade), or forging documentation.

One of the largest risks with counterfeit components is that they almost work correctly. Take this example of counterfeit electrical safety outlets - a component you probably don't think twice about:

Authorities in Suffolk County, N.Y. seized counterfeit electrical safety outlets—used in bathrooms, kitchens, and garages to guard against electrical shock—bearing phony UL logos. The bogus parts had no ground-fault-interrupt circuitry. Had they been installed anywhere near water, the results could have been fatal.

Many counterfeit components do not have such egregious and easily detected problems such as missing circuitry. Instead, the electrical characteristics of counterfeit components such as slew rate, current supply, timing, or noise might be out-of-spec. They also tend to be less reliable and exhibit a shorter time-to-failure than their legitimate counterparts. Counterfeit components can wreak havoc on consumer electronics. It would be utterly detrimental if they were to sneak into safety-critical devices like fire alarms, medical devices, or automotive electronics.

Detecting Counterfeit Components

Luckily, those who make counterfeit components are often not very good at it. Legitimate component manufacturers have high quality standards for their parts. In many cases, counterfeit components expose themselves with major packaging flaws. Common visual inspection cues are:

  • Incorrect part numbers
  • Incorrect date codes
  • Impossible date codes
  • Date codes that are in the future
  • Incorrect manufacturer country of origin marking
  • Components with the same lot code shown as being manufactured in different countries
  • Pre-soldered pins
  • Pins pitch is too wide or too narrow
  • Package made with the wrong material
  • Different numbers, shapes, and sizes of IC package indents
  • Laser cut lines in the markings
  • Incorrect font
  • Crooked or misaligned text
  • Incorrect silkscreen on a flexible circuit or PCB
  • Incorrect / incomplete logos
  • Logos that vary from part-to-part
  • Misspellings
  • Using ink-based IC markings that can be removed with acetone

Counterfeit components also give themselves away when comparing die shots between suspected counterfeits and known-good parts. Sometimes counterfeiters do a decent job with package markings, which may cause parts to slip through a visual inspection. Consider this example of a counterfeit Nordic NRF24L01+ transceiver. When the dies between the suspect parts and the legitimate parts are compared, you can clearly see that they are different. Unfortunately, capturing and comparing die shots safely requires the help of a lab.

For more information on spotting counterfeit components (including examples):

DARPA Anti-Counterfeiting Initiatives

The United States Department of Defense (DoD) has taken significant interest in counterfeit components. Most of the components used in military systems are produced outside of the US, where the DoD cannot regulate or influence off-shore IC fabrication. DARPA is sponsoring two anti-counterfeiting programs in order to combat the risk of out-of-spec, unreliable, and counterfeit parts: Integrity and Reliability of Integrated Circuits (IRIS), and Supply Chain Hardware Integrity for Electronics Defense (SHIELD).

The objective of IRIS is to develop techniques for non-destructive IC analysis. One major effort is the Advanced Scanning Optical Microscope (ASOM). ASOM enables researchers to scan ICs and provide sub-micron structural details. The IRIS program also aims to develop modeling and diagnostic techniques to determine the reliability of an IC.

SHIELD is focused on developing a small (100 micron x 100 micron) component that will authenticate electronic components. This component, called a dielet, will be inserted into IC packages by the manufacturer, but will require no electrical connection between the dielet and the host component. The final goal is that the authenticity of a chip can be confirmed on receipt by using a handheld or automated probing device.

For more on the DARPA anti-counterfeiting initiatives:

Using Blockchains

Proposals have started to appear for using blockchains to create a chain-of-custody for electrical components through the supply chain. The key features that blockhains can provide are non-localization, security, and auditability. A public, distributed resource prevents bad actors from hiding information and also helps create a publicly auditable system. Security is controlled through cryptography, which reduces the risk of forgeries and ensures that you can only interact with an account if you possess the correct key. Operations can be replayed and audited by anyone who joins the blockchain network since each transaction is permanently stored in the blockchain.

I think that creating a blockchain-based auditable chain-of-trust is a promising anti-counterfeiting measure despite the current excessive hype around them.

For more on using blockchains to fight counterfeiting:

What You Can Do Today

The best thing you can do to prevent the use of counterfeit parts is to always purchase your components from the original manufacturer or authorized distributors. Reputable suppliers do not want to risk their reputation by supplying bad parts and they do put effort into protecting their supply chain against infection. Buying components from eBay, discount retailers, or electronics markets increases the risk of receiving counterfeit components or recovered scrap material.

A secondary line of defense is incoming product inspection. Components and assemblies should be audited to ensure that counterfeit components are not being used. The inspection can be performed in-house or by a certified lab. If your product does not have an inspection process in place, this paper provides a basic inspection protocol.

We can also significantly reduce the risk of using counterfeit components by changing our expectations. When we apply cost and schedule pressures, engineers and manufacturers will cut corners and purchase from untrustworthy suppliers. By planning for sufficient lead times and paying manufacturer suggested prices we can decrease the likelihood of receiving counterfeit components.

For more on component counterfeiting:

Around the Web

We're already facing component shortages for OLED displays, DRAM, and NAND. It seems that we're likely to face longer lead times for IC packaging in 2018

Amazon announced its stewardship over the FreeRTOS kernel. The new kernel update sports AWS IoT integration right out of the box.

AllAboutCircuits recently posted GPS Times, Atomic Clock Frequencies, and the Increasing Accuracy of GPS. The article gives a quick intro to GPS time and the precision improvements that have been made over the past decades.

Website Updates

  • The Glossary has been reorganized and expanded
  • The Beginners page has new "General Resources", "Hardware" and "Startup" sections
  • The Hardware References page has been expanded with additional resources, including references on counterfeit component detection
  • The Software References page has been updated with more C++ blogs, a list of embedded systems newsletters, general software references, and Jenkins references

Articles Published in December

These posts were added to the website in December:

These were the most popular articles in December:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. std::string vs C-strings
  4. Implementing Malloc: First-fit Free List
  5. An Overview of C++ STL Containers
  6. A GitHub Pull Request Template for Your Projects
  7. Creating and Enforcing a Code Formatting Standard with clang-format
  8. C++ Casting, or: Oh No, They Broke Malloc!"
  9. memset, memcpy, memcmp, memmove
  10. An Improved Jenkins SCM Sync Configuration Plugin

Thanks for Reading!

Have any feedback, suggestions, interesting articles, or resources to recommend to other developers? Let me know!

Happy hacking!