September 2019: State Machines and Embedded.fm

Welcome to the September 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • State machine resources, libraries, and tooling
  • Embedded.fm podcast's huge milestone
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

State Machine Resources

This month we received multiple questions about state machines. State machines are an excellent area to study for embedded systems developers. Many embedded systems (or subsystems) can be modelled as state machines

If you're new designing state machines, you can learn more about them from expert Miro Samek and others:

We’re often asked about our favorite state machine code. Here are some of the best libraries:

We also answered questions this month about tools which can be used to model, verify, and generate code for state machines:

  • QM supports visual state machine modeling and code generation
    • The output code is targeted for the QP framework
  • Yakindu can model, verify, and generate code for state machines
    • The output code is not targeted to a specific library or framework
  • StarUML, PlantUML, and many other UML tools allow you to create visual models

We'll be turning this section of the newsletter into an article later this month. If you have any recommendations for state machine resources, libraries, or tooling, we'd love to hear from you.

Congratulations to Embedded.fm for 300 episodes!

Embedded.fm, our favorite embedded systems podcast, just released their 300th episode! We want to say a heartfelt congratulations to Chris and Elecia. It takes an enormous amount of dedication and persistence to create such a long-running show.

To celebrate their milestone, Chris and Elecia are throwing a party on September 7 in Aptos, California. If you're interested in attending, you can RSVP here.

If you haven’t heard of the Embedded.fm podcast, you can tune in at https://embedded.fm or in your favorite podcast app. Chris and Elecia are both embedded systems engineers. They started Embedded.fm to support engineers, encourage diversity, celebrate the ethics of good design, and help demystify what engineers do. They also wanted to show how a career in engineering can evolve. With 300 episodes under their belt, they have definitely stayed true to these goals.

A lot of time and energy goes into each episode. We encourage you to support the podcast through Patreon or by purchasing Embedded.fm merchandise. As a bonus to Patreon sponsors, you get access to a Slack group where you can interact with other embedded engineers and past podcast guests. We've found this group to be an extremely valuable resource for experiencing new perspectives and expanding our knowledge of embedded systems.

There have been so many wonderful episodes. Here is a small sampling of our favorites:

Around the Web

We’ve grouped our September reading recommendations by these categories:

  • Programming
  • Embedded
  • Hardware

Programming

John Regehr published a thorough guide describing how to write fuzzable code. If you're interested in applying randomized testing to your software, this is a great start.

All About Circuits published an introduction to virtual memory.

Burkhard Stubert discussed how data-driven unit tests are hard to understand.

Embedded

Mohammad Afaneh published a detailed Bluetooth Low Energy Primer on the Interrupt blog.

The Memfault team expanded their Zero to main() series with notes on how to write a bootloader from scratch.

Steve Branam updated his excellent So You Want to Be An Embedded Systems Developer article with new resources.

Niall Cooling, of Feabhas, shared two articles with us from his archives: Setting up the Cortex-M3/M4 MPU, and Developing a Generic Hard Fault handler for ARM Cortex-M3/M4.

Confused by the differences in SPI operational modes? Stratify Labs has a helpful visual guide.

Qt will be launching a bare metal toolkit targeted for microcontrollers.

Memfault posted an approach to building a CLI for firmware projects using the Python Invoke package.

Embedded.com featured an article discussing a logical method for debugging embedded systems.

Stratify Labs provides a straightforward strategy for reading floating pins.

Jacob Beningo published two articles about using JSON in embedded systems: Geetting Started with JSON for Embedded Developers and Using strstr to parse JSON Messages.

Embedded.com featured an article by Jacob Beningo describing three device driver design techniques.

Hardware

All About Circuits discussed tools and practices that can be used to reduce errors in your schematics.

This article surveying sub $0.10 microcontrollers has been circulating around Twitter and provides a look at the cheapest end of the microcontroller spectrum. The article was inspired by an article published last year: Making a Three Cent Microcontroller Useful.

Stratify Labs is publishing a series of articles to teach you the EE concepts needed for working with microcontroller-based circuits. The first three articles in the series are Ohm's Law for Makers, Kirchoff's Law for Maker's, and Equivalent Circuits for Makers.

All About Circuits shared an introduction to vibration energy harvesting as a source of power for electronics.

Job Postings

Apple, Inc.'s Hardware Test Engineering team is looking for an Embedded Systems Software Architect to lead the creation of a modular instrumentation platform. Are you convinced that with the right resources you could build a platform that will change an entire industry? Are you able to provide technical leadership across both the hardware and software realms? If so, contact Wei Wang to find out more.

At IRNAS, innovation lab for hardware and IoT products, based in Maribor, Slovenia, we are expanding our crew with an Embedded Software Engineer, who will collaborate on interdisciplinary development of telecommunication solutions, IoT systems, 3D bio-printing and more. Find out more about the open job position and apply by visiting this link.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our next newsletter.

Website Updates

We expanded the Glossary with additional terms and links. We also fixed the "D" section, which had duplicate entries.

An RSS link has been added to both the blog side panel and newsletter archive side panel.

New Articles

We published the following articles in August:

These were our most popular articles in August:

  1. Creating a Circular Buffer in C/C++
  2. Demystifying Microcontroller GPIO Settings
  3. Jenkins: Configuring a Linux Slave Node
  4. For Beginners
  5. Mixing C and C++: extern C
  6. std::string vs C-strings
  7. C++ Casting, or: "Oh No, They Broke Malloc!"
  8. Installing LLVM/Clang on OSX
  9. An Overview of C++ STL Containers
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip & Rozi

August 2019: Apollo 11, Design for the Blind, Architecture

Welcome to the August 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Apollo 11: 50th anniversary
  • Electronics design for the blind
  • Ravioli, a tool for calculating the Koopman Spaghetti Factor
  • Embedded systems architecture resources
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

50th Anniversary of Apollo 11

July marked the 50th anniversary of the Apollo 11 mission, which has garnered much media attention and the launch of the spectacular Apollo 11 in Real Time project. While it's interesting to revisit the Apollo 11 program, we're going to focus on a different angle: the Apollo Guidance Computer (AGC).

Space programs serve as excellent examples for embedded system designs with extreme requirements.

It's common to see quotes about the AGC like this one from Michio Kaku:

“Today, your cell phone has more computer power than all of NASA back in 1969, when it placed two astronauts on the moon.”

But statements like these are misleading. In an age of room-sized computers, the 70-pound Apollo Guidance Computer was an engineering marvel. The computer was responsible for computing the navigational course to the moon and adjusting that course based on in-flight astronaut input. The AGC needed to communicate with 150 different devices onboard the spacecraft - an immense task even for today's systems. The computer enabled Apollo 11 to get to the moon, reach the surface, and return to earth autonomously. And after a quarter of a million miles - the AGC hit its target within 500-600 feet of the goal. Such a feat of engineering is worth studying, which is why we feature the Apollo 11 AGC in our Source Code Classics article. For a book on the subject, check out Apollo Guidance Computer: Architecture and Operations.

Also, while we're all thinking about the Apollo programs, we want to share this wonderful anecdote from Margaret Hamilton:

Often in the evening or on weekends I would bring my young daughter, Lauren, into work with me. One day, she was with me when I was doing a simulation of a mission to the moon. She liked to imitate me – playing astronaut. She started hitting keys and all of a sudden, the simulation started. Then she pressed other keys and the simulation crashed. She had selected a program which was supposed to be run prior to launch – when she was already “on the way” to the moon. The computer had so little space, it had wiped the navigation data taking her to the moon. I thought: my God – this could inadvertently happen in a real mission. I suggested a program change to prevent a prelaunch program being selected during flight. But the higher-ups at MIT and NASA said the astronauts were too well trained to make such a mistake. Midcourse on the very next mission – Apollo 8 – one of the astronauts on board accidentally did exactly what Lauren had done. The Lauren bug! It created much havoc and required the mission to be reconfigured. After that, they let me put the program change in, all right.

For more on the Apollo 11 50th anniversary and the Apollo AGC:

Electronics Design for the Blind

IEEE Spectrum recently published an article titled "You Don’t Need Sight to Read These Electronic Schematics". Our interest was immediately piqued. The article briefly detailed developments in enabling electronic design for the blind. Electronics work seems like an area where sight is critical. It is exciting this assumption has been proven false.

The article describes Lauren Race's thesis project, Designing Tactile Schematics, which researched standards and best practices for designing tactile schematics. Race initially tried to print schematics on Swell Touch Paper, which causes areas printed in black ink to swell up and create raised surfaces when passed through a fuser machine. She learned through usability testing that making existing schematics tactile did not make them readable. She then worked with user groups to modify symbols and create schematics that were readable to both sighted and blind people. You can find the culmination of her work in her Visual Style Guide, as well as a sample set of 50 schematics following the tactile style guide rules.

The article also mentions the Blind Arduino Project, which aims to overcome the barriers faced by blind people who want to participate in hardware prototyping. The Project offers occasional hands-on workshops teaching coding, soldering, and other skills to blind makers. They also document their learnings on the Blind Arduino blog and host a regular Bay Area Blind Arduino Monthly Meetup.

For more on enabling electronics design for the blind:

Ravioli: A Tool for Calculating the Koopman Spaghetti Factor

Matt Chernosky, who runs Electron Vector, has created a software quality tool dubbed Ravioli. This tool calculates the Koopman Spaghetti Factor, a software complexity metric created by Phil Koopman.

The Spaghetti Factor (SF) metric is calculated in the following manner:

SF = SCC + (Globals*5) + (SLOC/20)

Where:

  • SCC = strict cyclomatic complexity (for more information, see the Phil's article).
  • Globals = number of read/write global variables, excluding const variables and file-scope statics
  • SLOC = source lines of code, which focuses on actionable programming statements and excludes comments

Phil Koopman provides a helpful and actionable scoring guide. Here are Phil's thoughts on complexity scores:

  • 5-10 - This is the sweet spot for most code except simple helper functions
  • 15 - Don't go above this for most modules
  • 20 - Look closely; review to see if refactoring makes sense
  • 30 - Refactor the design
  • 50 - Untestable; throw the module away and fix the design
  • 75 - Unmaintainable; throw the module away; throw the design away; start over
  • 100 - Nightmare; probably you need to throw the whole subsystem away and re-architect it

Our complexity analysis tool of choice is Lizard, but we will be adding Ravioli to our CI suite for another perspective. Ravioli is currently targeted for C source files, but we think it can also be adapted for C++.

For more on Ravioli and the Spaghetti Factor:

Embedded Systems Architecture Resources

Last month we received multiple requests for resources to help learn about embedded systems security. This month we had many engaging conversations about architecting embedded systems, which surprised us because "architecture" and "design" are typically words that send programming teams running for the hills.

These conversations prompted us to share our collection of embedded systems architecture resources. Software architecture is a broad field, so we've focused our recommendations on a the following areas:

We also frequently write about systems thinking and software architecture on Embedded Artistry, and we will be publishing more articles on these topics as we develop our architectural skills.

Around the Web

We've grouped our August reading recommendations by these categories:

  • General
  • Embedded
  • Security
  • Hardware

General

The authors of Elements of Programming have released a free PDF copy of the book in honor of the 10 year anniversary. This is a programming classic, so definitely take this opportunity to read it.

All About Circuits published a great article discussing the Principles of Cache Design, a topic low-level programmers should be familiar with.

Memfault published an article describing how to get the most out of the linker map file.

Memfault also published an excellent tutorial on automating debugging tasks with GDB's Python API.

Hillel Wayne published an approach for identifying missing requirements using modeling.

Jonathan Boccara, author of the Fluent C++ blog, published a wonderful post addressing a common problem we see: duplicate global data. Check out his article, What Every C++ Developer Should Know to (Correctly) Define Global Constants.

Embedded

Jay Carlson authored an excellent post describing how he teaches embedded systems to EE undergraduate students. Jay's notes are valuable for anyone who teaches embedded systems topics to others.

Elecia White shared a GitHub repository, Machine Learning on Embedded Devices, which serves as a wonderful collection of resources for those interested in the intersection of ML and embedded systems.

Security

Researchers seem to have found a way around Bluetooth LE's MAC address randomization feature, which is used to prevent identifying and tracking devices.

Memfault published Fix Bugs and Secure Firmware with the MPU, a guide for securing ARM MCUs with the Memory Protection Unit.

All About Circuits looked at microcontrollers with security-focused hardware components in Secure Microcontrollers Address IoT Security

All About Circuits highlighted Hardware Security Vulnerabilities that Engineers Should Know. The article concludes with advice for securing embedded systems.

Hardware

EDN published a review of TI's new tutorial book titled Radiation Handbook for Electronics. The book is pitched as "a compendium of radiation effects topics for space, industrial and terrestrial applications." Checkout this book if you're interested in designing electronics for high-radiation environments.

All About Circuits published an introductory article on Understanding Schematics, a critical skill for all embedded engineers.

Job Postings

At IRNAS, innovation lab for hardware and IoT products, based in Maribor, Slovenia, we are expanding our crew with an Embedded Software Engineer, who will collaborate on interdisciplinary development of telecommunication solutions, IoT systems, 3D bio-printing and more. Find out more about the open job position and apply by visiting this link.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our newsletter.

Website Updates

We expanded the Glossary with additional terms and links.

We updated Technology Radar statuses for items that were under evaluation.

We added additional Apollo 11 links to Programmers: Let's Study Source Code Classics.

New Articles

We published the following articles in July:

These were our most popular articles in July:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. Demystifying Microcontroller GPIO Settings
  4. For Beginners
  5. Installing LLVM/Clang on OSX
  6. Migrating from C to C++: NULL vs nullptr
  7. C++ Casting, or: "Oh No, They Broke Malloc!"
  8. An Overview of C++ STL Containers
  9. What can software organizations learn from the Boeing 737 MAX saga?
  10. Jenkins: Running Steps as sudo

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip & Rozi

July 2019: Security, Security, Security!

Welcome to the July 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Phillip's interview on Embedded.fm
  • Atomic clocks in space
  • Stuxnet: ten years later
  • Embedded systems security resources
  • Embedded news from around the web
  • Updates to the Embedded Artistry Website

Embedded.fm Interview

Last month, Embedded.fm interviewed Phillip. Elecia, Chris, and Phillip discussed consulting, writing, learning, code reviews, rules of thumb, Boeing 737 MAX, and implementing C++ threading types with an RTOS.

Check out Embedded.fm episode 290!

Atomic Clocks in Space

We love learning about technological developments with space exploration, since they involve wonderful examples of highly reliable embedded systems.

In June, NASA’s Jet Propulsion Laboratory (JPL) announced that they were creating a deep-space atomic clock. This clock is a toaster-sized device that aims to facilitate navigational efforts for future space missions. The first clock was launched into Earth’s orbit at the end of June for a one-year trial.

You can learn more about the Deep Space Atomic Clock with these JPL links:

A Look at Stuxnet, Ten Years Later

June marked the 10th anniversary of the first Stuxnet infection, which targeted Siemens PLCs and was responsible for damaging Iran's nuclear program. Stuxnet caused centrifuges at the nuclear plant to spin out of control while operator screens reported nominal values, leading to systems failures, asset damage, safety concerns, and a national security fiasco.

It's interesting to look back on Stuxnet with the context of today's continual onslaught of security vulnerability announcements and IoT botnet attacks. What have we learned in the ten years since Stuxnet demonstrated the capability of vulnerabilities to wreak havoc on physical systems? The overall state of device security hasn't noticeably improved, and reports of hacked devices reaching into the physical world abound.

Security can no longer be an afterthought. Even if your product isn’t connected to a nuclear reactor, vulnerabilities are exploited with disastrous results. For more on Stuxnet:

For more on Stuxnet:

Embedded Systems Security Resources

A reader reached out to us on LinkedIn and requested recommendations for books and resources discussing embedded systems security. As we just mentioned, security is an area that we all need to focus on when we design and build our embedded systems. If you have recommended books or links that discuss embedded systems security, let us know! We will be collecting recommendations and publishing a post dedicated to security resources. Here’s what we’ve collected thus far.

Phil Koopman has a collection of lectures discussing embedded systems security:

The following books relate to embedded systems security:

Here are articles with general introductions to embedded systems security concepts:

These articles focus on common pitfalls with embedded device security:

Around the Web

We've grouped our July reading recommendations by these categories:

  • General
  • Software
  • Security
  • Hardware

General

Elecia at Embedded.fm wrote a great post about Giving Feedback prompted by our conversation on the podcast.

Bunnie comments on the trade war with China and its potential impact on Open Source Could Be a Casualty of the Trade War.

All About Circuits published a helpful introduction to the Advanced Microcontroller Bus Architecture (AMBA), which is the de facto standard for SoC design.

Software

If you're interested in the intersection of machine learning and embedded systems, a reader on LinkedIn shared this link for running TensorFlow Lite on a microcontroller.

The Memfault team published an excellent article on digging into firmware code size. If you're trying to reduce your binary's size, this article is a must read.

The Memfault team also published an excellent article on writing linker scripts for firmware.

Colin Walls wrote about mixing C and C++ in an embedded application. If you're interested in trying out C++ on an existing embedded product, try one of Colin's two approaches.

Burkhard Stubert added another article to his Speaking CAN series, this time analyzing write buffer overflows.

Watchdog timers are a staple element of embedded systems. Alexandru Lazar shared some Watchdog Timer Anti-patterns that we should keep in mind when building our systems.

Thread Group announced version 1.2 of the Thread wireless protocol. Stacey Higginbotham provided a summary of the changes

For a great example of applying std::variant to create a state machine, check out this C++ article: Space Game: A std::variant-Based State Machine by Example.

Security

Appropriately for the 10-year anniversary of Stuxnet, we saw an inordinate amount of articles related to security.

EE Times highlighted that the IoT security gap is widening: we're adding more devices and sending more data, but IoT devices are still a prime hacking target.

Designing Hardware for Data Privacy highlights the need to develop a thread model for our devices and to implement the appropriate countermeasures.

Enhancing Privacy and Security in the Smart Meter Lifecycle provides a look at a variety of techniques that can be used to improve IoT device security.

How Many Layers of Security Do You Have? argues that designers should consider two guiding principles: defense in depth and principle of least privilege.

Will the Real Root of Trust Stand Up? discusses the concept of a "root of trust".

Hardware

Strange Parts recorded a fantastic factory tour of the PCBWay factory in Shenzen: Inside a PCB Soldering Factory - in China. If you are curious about how circuit boards are manufactured, check out this video.

Dangerous Prototypes shared a helpful app note discussing magnetometer placement in mobile devices.

Dangerous Prototypes also shared a helpful app note discussing a method for using two accelerometers to make rotational measurements, which is useful if you already have two accelerometers and don't want to update your product to add a gyroscope.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our newsletter.

Website Updates

Stickers are now available in the Embedded Artistry store. You can help support the website by purchasing a sticker.

We've renamed the "Embedded C++" and "Migrating from C to C++" categories to work around a Squarespace link generation issue.

We added additional acronyms to the Glossary.

We added new book recommendations and links to For Beginners, Hardware References and Software References.

We updated the following articles:

New Articles

We published the following articles in June:

These were our most popular articles in June:

  1. Creating a Circular Buffer in C/C++
  2. For Beginners
  3. std::string vs C-strings
  4. Demystifying Microcontroller GPIO Settings
  5. Installing LLVM/Clang on OSX
  6. C++ Casting, or: "Oh No, They Broke Malloc!"
  7. Improving Your Callback Game
  8. Jenkins: Configuring a Linux Slave Node
  9. An Overview of C++ STL Containers
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip & Rozi

June 2019: Programmer vs Engineer, Coroutines, Report CI

Welcome to the June 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Programming vs engineering
  • Coroutines
  • Report CI
  • Embedded news from around the web
  • Updates to the Embedded Artistry Website

We also want to share some exciting news with you! One of our articles was published in the June 2019 issue of the Software Quality Professional Journal. You can read The Boeing 737 MAX Saga: Lessons for Software Organizations for free if you register for an account with ASQ. Or you can email us and we will send you a copy.

Programmer vs Engineer

We recently came across an article from The Atlantic titled "Programmers: Stop Calling Yourselves Engineers".

The article begins by highlighting the use of "engineer" for programming roles:

In the Silicon Valley technology scene, it’s common to use the bare term “engineer” to describe technical workers. Somehow, everybody who isn’t in sales, marketing, or design became an engineer. “We’re hiring engineers,” read startup websites, which could mean anything from Javascript programmers to roboticists.

What's wrong with programmers co-opting the term "engineer"? The article points out that:

Traditional engineers are regulated, certified, and subject to apprenticeship and continuing education. Engineering claims an explicit responsibility to public safety and reliability, even if it doesn’t always deliver.

The fact that programmers don't have a similar set of responsibilities is alarming because:

Computing has become infrastructure, but it doesn’t work like infrastructure.

And aside from computing infrastructure that we've come to rely on, we are building more and more software-intensive complex systems:

It’s no accident that the most truly engineered of software-engineering projects extend well beyond the computer. Autonomous-vehicle design offers the most obvious contemporary example. When Google designs self-driving cars, it musters its own computational systems, like mapping and navigation. But it also integrates those into a world much larger than browsers and smartphones and data centers. Autonomous vehicles share the roads with human-driven cars, pedestrians, and bicyclists. Those roads are managed, maintained, and regulated. Self-driving cars also interface with federal motor-vehicle standards and regulations, along with all the other material demands and foibles of a machine made of metal and plastic and rubber rather than bits. Engineering addresses complex, large-scale systems.

We're sympathetic to the article. The fundamental complaint seems to be that companies are co-opting the term engineer for glory, but they are merely profiteering raiders who are avoiding the rigor required of engineering professionals. We can't deny this assessment given many of the requests we receive from new hardware companies looking to strike it rich as quickly as possible.

Fortunately, there seems to be a slowly growing wave of increasing professionalism that may stand to counter the reckless creation of software systems. Jack Ganssle tells a chilling tale about the evolution of fire safety standards and equates the situation to software. He, and many others, also call for eliminating the term "bug" and replacing it with "error".

Jack’s fundamental question still rings true, especially in light of the Boeing MCAS debacle and loss of 346 lives. How many more software-related catastrophes will we experience before we demand a fire code for people working on critical software systems and infrastructure? If programmers want to be considered engineers they ought to be held to the same standards as engineers in other specialities.

Coroutines

We've recently been digging into coroutines for our C++ framework, which is well-timed because they are slated to land in C++ with the upcoming C++20 release. Although they are just being added to C++, the coroutine concept exists in a variety of languages and can be applied even without language standard support (see libaco for a C implementation, and Klemens Morgenstern's talk for a lightweight implementation for C++ on ARM controllers.

As a brief overview, a coroutine is a function which:

  • May or may not have its own stack (depending on the implementation)
  • Can interrupt its own execution at specific suspension/yield points
  • Can be resumed later

Coroutines provide an asynchronous processing alternative to threads. Instead of having a scheduler which interrupts threads, coroutines interrupt themselves. Asynchronous processing is achieved by manually sequencing slices of coroutine functionality.

As always, there are tradeoffs: coroutines give us deterministic asynchronous processing, but they do not provide true concurrency that can take advantage of multiple processors or cores. This set of tradeoffs is generally attractive for embedded systems, as embedded systems designers are often more concerned with determinism than taking advantage of concurrent processing.

The links we list below have helped us immensely in our study of coroutines. We'll be applying them to our embedded C++ framework to simplify some complex areas of asynchronous processing that currently uses hard-to-follow chains of callbacks (these notes from IT Hare demonstrate the complexity problems we are facing). Avoiding chains of callbacks will greatly simplify our asynchronous code.

For more on coroutines in general:

For more on C++ coroutines:

Report CI

Klemens Morgenstern has released a new GitHub integration called Report.ci. Report.ci allows you to report automated unit test results and compiler logs to GitHub from your chosen continuous integration (CI) system. Rather than having to dig around in the build server logs to figure out why a build or test failed, the results can be uploaded to Report.ci and shown directly in the GitHub pull request (under both "Checks" and "Files Changed").

Here are some examples of the service in action:

Report.ci currently supports 28 unit test frameworks and compiler log annotations for many popular languages and compilers. Other capabilities of the service include Badges, Annotation merging for cross-compilation scenarios, and mapping annotations so they appear at the code under test (rather than the test code itself).

Report.ci is free for public GitHub projects to use. Give this plugin a try if you want to see test results and compiler logs directly within your GitHub pull requests.

Around the Web

We've grouped our May reading recommendations by these categories:

  • Software
  • Hardware
  • History

Software

Niall Cooling continued his series on the difference between semaphores and mutexes with Part 2: The Mutex & Mutual Exclusion Problems.

François Baldassari, CEO of Memfault, shared his take on the program startup process in From zero to main(): Bare metal C.

Mohammad Afaneh of NovelBits published a guest post on the Interrupt blog on Debugging Firmware with GDB. If you are unfamiliar with using GDB,

Fotis Chatzinikolaou shared helpful communication protocol design tips in Designing Communication Protocols, Practical Aspects.

Hardware

Maya Posch published an article on Hackaday which describes the physics behind soldering and IC packaging.

Arch Martin at AVX wrote an excellent app note describing the basics of decoupling.

We found an app note from Renesas on eye safety protection guidelines for products using infrared LEDs.

History

Take a look at the die of Intel's 8087 floating point coprocessor chip

All About Circuits published an article describing the early history of the oscilloscope.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our newsletter.

Website Updates

We added new links to the Software References, Hardware References, and For Beginners pages. The new links include a list of all Embedded.fm episodes, links to video playlists, and a review of free Python courses.

A link to our curated YouTube video playlists has been added to the "Resources" menu.

We corrected typos in our article: Ditch Those Built-In Arrays for C++ Containers.

New Articles

We published the following articles in May:

These were our most popular articles in May:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. Installing LLVM/Clang on OSX
  4. Demystifying Microcontroller GPIO Settings
  5. Migrating from C to C++: NULL vs nullptr
  6. C++ Casting, or: "Oh No, They Broke Malloc!"
  7. An Overview of C++ STL Containers
  8. Jenkins: Configuring a Linux Slave Node
  9. Jenkins: Running Steps as sudo
  10. Improving Your Callback Game

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

May 2019: GPS Rollover Day, Historical Source, and YouTube Playlists

Welcome to the May 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • GPS rollover day problems
  • Historical source code
  • Embedded Artistry YouTube playlists
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

GPS Rollover Problems

6 April 2019 was "GPS Rollover day". The current GPS standard uses a 10-bit counter to keep track of the number of weeks elapsed since 6 January 1980. Since it's a 10-bit counter, the max count is 210-1, or 1023, weeks. On the 1024th week, it will rollover to week 0.

If you’re keeping score, you’ll be surprised that this was a big deal at all: we already experienced a GPS rollover on 21 August 1999. Many GPS equipment manufacturers understand the rollover requirements and designed their systems to accommodate it. However, we knew that some GPS receivers would be unable to make the transition gracefully. These receivers would reset their date to the last epoch (21 August 1999). Once the epoch is reset, these devices were expected to give the wrong time and position. In the worst case devices might lock up permanently.

Companies had plenty of time to prepare for this event and firmware updates were sent out for many affected devices. But as expected, GPS Rollover Day still impacted many systems:

Let this be a warning to system designers: your systems will remain in use longer than you expect. Spend time investigating and handling edge cases, especially foreseeable ones like a recurring 1024-week GPS rollover.

Historical Source Code

Contemporary programmers are lucky: we live in a world where historical and influential program source code is available for us to review. However, most programmers only learn and study the programs they have worked on themselves. We rarely take the time to study historical works, and programming courses don’t typically spend any time on the subject.

We believe that software developers should review influential source code. This is similar to architects studying influential building designs (and critiques of those designs). Rather than repeating the same mistakes over-and-over, we should study the great works that preceded us and learn from and build upon their lessons.

If you're interested in studying historical source code, we've collected links in a new article on our blog: Programmers: Let's Study Source Code Classics.

If you have recommendations for classic source code or commentary, simply respond to this email and we will add them to the list.

Embedded Artistry YouTube Playlists

Over the past three years, we've shared dozens of YouTube videos on Twitter and in our newsletter. Most of those videos were shared only once. If you weren't looking at just the right moment, you missed the link. We've decided to collect our video recommendations in a YouTube channel.

We've created the following playlists and populated them with videos we've recommended in the past:

There are already many quality embedded systems playlists on YouTube. You can find our playlist recommendations in the Saved Playlists view and on the channel home page.

We will feature other channels that regularly publish quality embedded systems content, such as:

We will be continually expanding these playlists as we discover new videos. If you have videos to recommend to the embedded community, simply respond to this email.

Around the Web

We've grouped our May reading recommendations by these categories:

  • Industry News
  • Product Development
  • Firmware
  • Hardware

Industry News

Microsoft bought ThreadX, which is our favorite RTOS. Now we see that major tech companies are individually sponsoring an IoT/RTOS solution: Microsoft has ThreadX, Amazon has FreeRTOS, Intel has Zephyr, and Google has Fuchsia.

The Zephyr Project made their first long-term support (LTS) release. Here is more information about the plans for Zephyr LTS:

Product Development

Not all signal processing applications need a fully-fledge Digital Signal Processor (DSP) All About Circuits provides helpful tips for selecting a microcontroller to use instead of a DSP chip for signal processing applications:

These relate to another helpful All About Circuits article published last month: How to Choose the Right Microcontroller for Your Application.

Firmware

All About Circuits published a high-level Introduction to Real-Time Embedded Systems

Niall Cooling cleared up common misconceptions about mutexes and semaphores in Mutex vs Semaphore: Pt 1.

David Bergfeld walks through a thorough state machine design exercise in Software Design of State Machines.

Avinash Aravindan concluded his Flash 101 series with Error Management in NAND Flash. If you haven't followed along, here are the first five articles:

  1. NAND Flash vs NOR Flash
  2. The NOR Flash electrical interface
  3. The NAND Flash electrical interface
  4. Types of NAND Flash
  5. Errors in NAND Flash

Colin Walls Some More Embedded Software Programming Tips

Hardware

Microcontroller datasheets are massive tomes which can intimidate new engineers. All About Circuits shared two helpful guides for navigating processor datasheets:

Lee Teschler at Microcontroller Tips discussed the Hidden Pitfalls of IoT Antenna Design

All About Circuits provided a helpful guide to Understanding Amplitude Quantization Error for ADCs.

EETimes gave us a look at the power systems engineering for satellites in Taking Care of Power Systems for Space Missions.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all our open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We expanded the Glossary with additional terms:

  • Bottom-half handler
  • Top-half handler
  • LMA
  • VMA
  • XIP

We expanded the For Beginners page with Miro Samek's Modern Embedded Systems Programming Video Course and links to help readers learn RTOS concepts.

We updated the following articles with corrections and new content:

New Articles

We published the following articles in April:

These were our most popular articles in April:

  1. What can software organizations learn from the Boeing 737 MAX saga?
  2. Creating a Circular Buffer in C/C++
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. Migrating from C to C++: NULL vs nullptr
  6. Demystifying Microcontroller GPIO Settings
  7. Jenkins: Configuring a Linux Slave Node
  8. An Overview of C++ STL Containers
  9. C++ Casting, or: "Oh No, They Broke Malloc!"
  10. Jenkins: Running Steps as sudo

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

April 2019: Boeing 737 MAX and Mirai

Welcome to the April 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The grounding of the Boeing 737 MAX
  • The return of the Mirai botnet
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

Grounding the Boeing 737 MAX

One of the largest news stories over the past month was the grounding of Boeing 737 MAX-8 and MAX-9 aircraft after an Ethiopian Airlines crash resulted in the deaths of everyone on board. This is the second deadly crash of involving a Boeing 737 MAX. A Lion Air Boeing 737 MAX-8 crashed in October 2018, also killing everyone on board. As a result of these two crashes, Boeing 737 MAX airplanes are temporarily grounded in over 41 countries, including China, the US, and Canada. Boeing also paused delivery of these planes, although they are continuing to produce them.

We've been following the Boeing 737 MAX story closely. It serves as an interesting case study on software and systems engineering, human factors, corporate behavior, and customer service.

We started writing about the Boeing 737 MAX investigation as part of the newsletter and ended up with an 8,000 word essay on the saga. We cover the problem as it is currently understood, discuss contributing factors which are common across engineering organizations, and focus on lessons we learned from researching the crashes. I hope that all of you will take the time to read the article. There are important lessons we can all learn from this tragedy.

*Note: Both the Lion Air and Ethiopian Airlines crashes are still under investigation. Ultimately, everything you are reading about these crashes and that we discuss in the article is still in the realm of speculation. However, the situation is serious enough and well-enough understood that Boeing is addressing the problem immediately.*

Mirai Returns

In 2016, the Mirai botnet was discovered after it was used in some of the largest DDoS attacks. You were probably impacted by Mirai when the Dyn attack rendered many popular sites inaccessible, such as GitHub, Twitter, Reddit, Netflix, Paypal, Fox News, CNN, and Amazon.

Mirai is malware that targeted networked IoT devices running Linux. Many companies ship devices with default usernames and passwords enabled. Mirai takes advantage of this fact by continuously scanning for vulnerable devices and using an expansive list of factory default logins. If successful, the victim’s IP and login credentials was sent back to a collection server. Devices remain infected until they are rebooted, but they are quickly re-infected if the login is unchanged.

While the original author and his compatriots pled guilty to the Mirai-related crimes, the Mirai source code was released and used by other groups.

A new variant was discovered in January and announced this month. This new variant includes new credentials and exploits. The target list has been expanded, encompassing routers, network storage devices, IP cameras, and TVs. WePresent WiPG-1000 Wireless Presentation systems and LG Supersign TVs, both of which are marketed to businesses, are two devices which are specifically targeted. Enterprises are juicy targets for botnets because they typically have access to larger bandwidths, which the attackers can use to increase the impact of DDoS attacks.

As embedded systems creators, the only thing we can do to help stem the flood of compromised devices and botnet attacks is to set our own houses in order. If you are building an embedded system, change the default passwords and ensure that you are applying security patches for your devices. If a device on your business's network has a security flaw or default login that cannot be changed, you must remove it or firewall it.

For more on the new variant of Mirai:

For more on the original wave of Mirai attacks:

Around the Web

There was a lot of activity in March, so we've grouped our reading recommendations by category:

  • Product Development
  • Firmware
  • Hardware

Product Development

Kerry Scharfglass gave an absolutely wonderful talk on Design for Manufacturing. Kerry shares many of the lessons he learned the hard way about factories and manufacturing test software. If you're working on an embedded device and haven't yet been to a factory, this is a great introductory lesson and can save you a ton of pain.

The Verge published a look at the original iPhone development kit. Apple prototype development boards were the best that I've ever used as a firmware engineer - use some of these design ideas for your next dev board.

The terrible nature of IoT security has prompted Congress to introduce a bill titled The Internet of Thing Cybersecurity Improvement Act. Lawmakers are looking to legislate a bare minimum security standard that all IoT devices would have to meet. Stacey Higginbotham shared her take on the IoT bill.

Nike's self-lacing shoes stopped working after an update, even with the manual buttons. This reminds us to be extremely careful when designing products that require an Internet connection to function.

Firmware

Burkhard Stubert took a small survey of companies running demos of neural networks on microcontrollers.

Neil Tan went further and wrote a guide for running a simple neural network on MCUs. Neil's guide will enable you to train and load models on a microcontroller.

Jacob Beningo published "Minimizing energy consumption in Amazon Free RTOS applications" on Embedded.com.

Piotr Grygorczuk shared his implementation of C++ threading using FreeRTOS (std::thread, std::mutex, std::condition_variable, std::chrono, std::sleep_for, etc.).

Rud Merriam tweeted to us about his Embedding C++ series on Hackaday, where he explores the use of C++ on embedded systems.

Andreas at AndWass C++ recently started his blog. He's published three great embedded-related C++ articles:

  1. Multitasking 1 of n
  2. A Basic Scheduler
  3. Exploring A Driver Concept

Hardware

ST Micro published a great technical note addressing frequently asked questions related to Reset and Supervisor ICs.

Not sure what IoT standard to choose for your product's radio? This article from Microcontroller Tips attempts to predict the winners of the IoT radio standards war.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all our open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We fixed broken links on our Resources for Beginners page.

We updated our Technology Radar with the latest changes in our technology stack.

We updated the following articles with new content:

New Articles

We published the following article in March:

These were our most popular articles in March:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. C++ Casting, or: "Oh No, They Broke Malloc!"
  4. Installing LLVM/Clang on OSX
  5. Jenkins: Configuring a Linux Slave Node
  6. Jenkins: Running Steps as sudo
  7. Migrating from C to C++: NULL vs nullptr
  8. Jenkins: Kick off a CI Build with GitHub Push Notifications
  9. An Overview of C++ STL Containers
  10. Demystifying Microcontroller GPIO Settings

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

March 2019: Deprecating Volatile

Welcome to the March 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • How we get to main()
  • Deprecating volatile
  • Volatile load/store
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

What Happens Before Main?

We're working on a new article which describes the steps our programs take to get to the main() function. Whether we are running our programs on a PC or an embedded system, there are plenty of behind-the-scenes steps needed to get our system into a usable state.

At a high level, Embedded Artistry systems tend to match the following general pattern:

  1. Initialize the processor & external memory
  2. Setup the C/C++ runtime (reloc, BSS, constructors)
  3. OS initialization
  4. Hardware platform initialization (configure IO, start drivers)
  5. Jump to main()

We would love to hear from you: what steps does your embedded system take to get to main()?

Your feedback will help us refine our general boot model.

Deprecating Volatile

A recent C++ proposal for deprecating the volatile keyword has surfaced. This may surprise our readers, because as Michael Caisse said, "volatile is the embedded keyword."

The original intent of the volatile keyword in C89 is to suppress read/write optimizations:

No cacheing through this lvalue: each operation in the abstract semantics must be performed (that is, no cacheing assumptions may be made, since the location is not guaranteed to contain any previous value). In the absence of this qualifier, the contents of the designated location may be assumed to be unchanged except for possible aliasing.

The problem with its use in C++ is that the meaning is much less clear, as it is mentioned 322 times in the current draft of the C++ Standard.

One problematic and common assumption is that volatile is equivalent to atomic. All the volatile keyword denotes is that the variable may be modified externally, and thus reads/writes cannot be optimized. This means that the volatile keyword only has a meaningful impact on load and store operations.

Where programmers run into trouble is using volatile variables in a read-modify-write operation, such as with the increment (++) and decrement (--) operators. These operations create a potential for a non-obvious race condition, depending on how the operation is implemented in the compiler and platform. Chained assignments of volatile values (a = b = c) are another problematic volatile scenario. Is b re-read before storing the value to a, or not?

To address these usage problems, the authors created a separate proposal for volatile_load() and volatile_store() functions. We suggest an initial implementation of these functions in the next section.

Here's the list of proposed changes with the "deprecation" of volatile:

  1. Continue supporting the time-honored usage of volatile to load and store variables that are used for shared memory, signal handling, setjmp / longjmp, or other external modifications such as special hardware support.
  2. Deprecate (and eventually remove) volatile compound assignment op=, and pre / post increment / decrement -- ++.
  3. Deprecate (and eventually remove) volatile-qualification of member functions. Don’t change volatile-qualification of data members.
  4. Deprecate (and eventually remove) partial template specializations involving volatile, overloads on volatile, and qualified member functions for all but the atomic and numeric_limits parts of the Library.
  5. Deprecate (and eventually remove) volatile member functions of atomic in favor of new template partial specializations which will only declare load, store, and only exist when is_always_lock_free is true. Preserve most volatile free function overloads for atomic.
  6. Deprecate (and eventually remove) non-reference and non-pointer volatile parameters. Deprecate (and eventually remove) constas well as volatile return values. References and pointers to volatile data remain valid.

A rationale for each of these is provided in the original paper, "§3.6 Why the proposed changes?".

For more on the proposals related to deprecating the volatile keyword:

A Volatile Load/Store Implementation

Prior to the proposal, Embedded Artistry implemented volatile_load<T>() and volatile_store<T>() template functions to encourage better volatile behavior in our programs.

Since developers commonly misinterpret volatile access as atomic, seemingly innocuous statements can have surprising effects on program behavior if you are depending on atomic access.

volatile int i = 2; //probably atomic
i++; //not atomic ...

Using volatile_load() and volatile_store() ahead of standards changes will force a better usage pattern:

auto r = volatile_load(&i);
 r++;
 volatile_store(&i, r);

You can use these functions to refactor your programs and control volatile use cases. While this implementation does not meet the proposed specification, it's a step toward cleaning up the volatile keyword.

Here is our implementation of volatile_load() and volatile_store():

#include <cassert>
#include <type_traits>

/** Read from a volatile variable
 *
 * @tparam TType the type of the variable. This will be deduced by the compiler.
 * @note TType shall satisfy the requirements of TrivallyCopyable.
 * @param target The pointer to the volatile variable to read from.
 * @returns the value of the volatile variable.
 */
template<typename TType>
constexpr inline TType volatile_load(const TType* target)
{
    assert(target);
    static_assert(std::is_trivially_copyable<TType>::value,
        "Volatile load can only be used with trivially copiable types");
    return *static_cast<const volatile TType*>(target);
}

/** Write to a volatile variable
 *
 * Causes the value of `*target` to be overwritten with `value`.
 *
 * @tparam TType the type of the variable. This will be deduced by the compiler.
 * @note TType shall satisfy the requirements of TrivallyCopyable.
 * @param target The pointer to the volatile variable to update.
 * @param value The new value for the volatile variable.
 */
template<typename TType>
inline void volatile_store(TType* target, TType value)
{
    assert(target);
    static_assert(std::is_trivially_copyable<TType>::value,
        "Volatile store can only be used with trivially copiable types");
    *static_cast<volatile TType*>(target) = value;
}

Around the Web

Our February reading recommendations are grouped by category as follows

  • General Software
  • Firmware
  • Hardware
  • Product Development
  • C++

General Software

John Regehr collected samples of ASCII art that explains source code. If you're interested in adding ASCII art to your own code documentation, try out Buttersquid.ink. Buttersquid provides a link in the generated text so you can make updates to the diagram in the future.

We've talked about Spectre and Meltdown in past editions of our newsletter. IEEE published an informative article reviewing how the two attacks actually work.

The "Small Change" Fallacy makes us think we can skip full validation because a change is "small". All changes, no matter how innocuous, deserve to be treated seriously by running through the full code change process.

Jack Ganssle republished an old April Fool's Day joke: "Goto considered needed'.

Firmware

Jack Ganssle discussed the impending failure of older GPS devices due to an overflow problem.

Segger has launched the Smash-V2 compression algorithm, which is targeted for embedded systems. Compression requires 900B of ROM, and decompression requires 500-800B. No working RAM is required. Information is available on the Segger website.

Dan Luu analyzed the comparative performance of least-recently-used (LRU) cache eviction and random cache eviction. It turns out that random eviction performs pretty well under certain scenarios.

Arduino launched their own IoT cloud platform.

Mohammad Afaneh published two articles on prototyping with the nRF52840 USB dongle:

All About Circuits has published a set of articles introducing processor architectures:

Hardware

Japanese scientists have developed a MEMS energy harvester which can turn vibrations into power.

"Ensuring Fail-safe Data Storage in Battery Powered IoT Sensor Nodes" provides a strategy to address a common concern in battery-powered embedded systems.

The race is on to find an alternative to SRAM due to energy inefficiencies that are highlighted by the increase in data that the computing industry is managing. Part one of the series describes some of the problems with SRAM.

EE Times discussed the US-China trade crisis and the fallout for the chip industry.

Not sure what the difference is between slew rate and rise time? EDN clarifies the two concepts in "Slew Rate and Rise Time: Not Quite the Same".

Product Development

Pete Staples of Blue Clover Devices was on The Hardware Entrepreneur podcast. He discussed pitfalls with the NPI process and their new production line tool.

Bunnie Huang shared a talk on Supply Chain Security, a topic we've discussed in a previous edition.

C++

Jonathan Müller put together a great reference for when you need to declare different special member functions.

Rainer Grimm, a C++ trainer, addresses common template misconceptions and surprises.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all of their open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We added additional links and book recommendations to both the Software References and Hardware References pages.

For Beginners was updated with additional links.

We updated the following articles with new content:

New Articles

We published the following article in February:

These were our most popular articles in February:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. Jenkins: Configuring a Linux Slave Node
  4. C++ Casting, or: "Oh No, They Broke Malloc!"
  5. Installing LLVM/Clang on OSX
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. Jenkins: Running Steps as sudo
  8. Migrating from C to C++: NULL vs nullptr
  9. Demystifying Microcontroller GPIO Settings
  10. An Overview of C++ STL Containers

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

February 2019: Bluetooth 5.1 and Peer Code Review

Welcome to the February 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The new Bluetooth 5.1 specification
  • Phil Koopman's peer code review best practices
  • Embedded news from around the web (there was a lot of activity this month!)
  • Embedded job postings
  • Updates to the Embedded Artistry Website

Bluetooth 5.1 Released

The Bluetooth SIG released Bluetooth version 5.1 in January. The Bluetooth 5.1 release adds new features and capabilities to Bluetooth Low Energy (BLE).

Direction finding is the primary feature of the Bluetooth 5.1 release. The new direction finding capabilities include both Angle of Arrival (AoA) and Angle of Departure (AoD) measurements. Location services, such as asset tracking, item location, proximity marketing, and indoor way-finding, will use the new BLE direction finding capabilities. For products to take advantage of the new direction finding features, multiple antennas are required: 2+ on receivers for AoA method, and 2+ on transmitters for AoD method. BLE stacks will need controller-level updates to support the new features.

Other features of the 5.1 release include:

  • GATT Caching enhancements to reduce connection times in unbonded devices
  • Randomized advertising channel indexing to improve packet collision avoidance
  • Periodic advertising synchronization transfer
  • HCI support for Debug Keys in LE secure connections
  • Sleep clock accuracy update mechanism
  • ADI field in scan response data
  • Interaction between QoS and Flow Specification
  • Host Channel Classification for Secondary Advertising
  • Allow the SID to Appear in Scan Response Reports
  • Specify the behavior when rules are violated

For more on Bluetooth 5.1:

Phil Koopman on Peer Code Reviews

Peer code review is a code quality practice that we encourage all teams to adopt. This month, we listened to Phil Koopman's lecture series on Peer Code Review. This lecture series is full of actionable information which can help teams looking to adopt a formal peer code review practice. The series also serves as a source of new ideas for teams already performing reviews.

One area we want to highlight is Phil's rules for successful peer reviews:

  • Review everything that your team writes down:
    • Documentation
    • High-level design
    • Detailed design
    • Code
    • Test plans
  • Inspect the item, not the author
  • Don't get defensive - nobody writes perfect code
    • If you are getting defensive as the author, leave the room
  • Find problems; don't fix them in the meeting
    • Identify the symptom or technical deficiency of the problem as specifically as possible
  • The word "should" is not appropriate in a peer review
  • No debates on style
    • Enforce conformance to style guide
    • Do not debate whether the style guide is correct

Phil expands upon these rules with his peer code review best practices:

  • Formal reviews (Fagan inspections) optimize bugs found per dollar
    • Formal reviews find more bugs than informal reviews, and they find them more efficiently
    • Informal reviews typically find < 10-50% of defects
    • Online-review tools are OK, but they are not substitute for formal in-person review meetings
  • Target 10% of total project effort to find 50% of bugs
  • Also review documentation and designs - not just code
  • Review documentation and design first to find issues before coding begins
  • Use a checklist to find the most bugs
  • Target 1 peer review per day so you don't burn out reviewers
  • Schedule peer reviews regularly
    • 2-3 reviews per week as the project progresses
  • Limit meetings to 1-2 hours
    • After 2 hours, review effectiveness drops off
  • Target 150 lines of non-comment source code per hour for review
  • Keep a queue of items pending review; pull off queue for a scheduled review
  • Run static analysis tools before review meetings
    • Static analysis tools do not eliminate the need for peer code review

Phil states that formal peer code reviews should capture > 50% of defects. If your team is not finding sufficient bugs during code reviews, listen to the lecture series and try applying a formalized review process. All of us can improve the quality of our peer code review by applying these straightforward rules and best practices in our organizations.

Here are additional publications on peer code reviews by Phil Koopman:

Here are Embedded Artistry publications discussing peer code reviews:

Around the Web

There was a lot of activity in January, so we've grouped our reading recommendations by category:

  • Firmware
  • Hardware
  • C
  • C++

Firmware

Jack Ganssle shared insights into how we can improve our firmware development processes.

Researchers in Japan studied how tree frogs regulate their communications to avoid croaking over each other frog in the colony. Their findings have implications for a future network packet collision prevention scheme.

Low-powered IoT devices with limited memory capabilities will have to eventually support IPv6. 6LoWPAN will make that happen.

Japan plans to hack 200 milion IoT devices in February to compile a list of vulnerable devices. This effort further highlights the fact that IoT devices, not people, are now the security weak link.

The Multicore Association has released a new version of the Software-Hardware Interface for Multi-Many Core (SHIM). Version 2 of the specification introduces new features around power consumption properties, inter-connect contention descriptions, custom instructions, vendor extensions, and more.

Hardware

EDN gave us a look at NASA Apollo electronics from the 1960s.

Samsung shipped the first 1 terabyte NAND chip.

For years we have faced a constrained DRAM supply with rising prices. Now we are dealing with over-supply and a forecasted 20% price crash.

Nanopower announced their nP-BLE52 module, which is a System-in-Package (SiP) combining an nRF52832 with a custom power management IC. The custom power management IC enables the SiP to put the SoC into sleep mode, cut power, and wake it up at a pre-set time and restore the pre-sleep state. Sleep mode current consumption is spec'd at 10nA, making this processor ideal for low-power and long-life embedded applications.

Apple has filed a patent for a new version iBeacon, which will utilize ultra-wide band radio instead of BLE.

The electronics industry shipped over 1 trillion devices last year, including integrated circuits and optoelectronics, sensors, and discrete (O-S-D) devices.

All About Circuits has been publishing reviews of sensors, in case you need inspiration for your next project:

C

Feabhas published an article describing How to use C structs to access memory mapped peripheral registers. Many vendor SDKs and frameworks use this technique for interacting with processor peripherals.

C++

C++ can be a confusing language, especially when details such as lvalues and rvalues come up. This practical post on C++ moves is for those developers who don't know (or don't care) what rvalues are.

C++11 added smart pointers to our arsenal, but it's not always clear how we should move them into and out of functions.

As is commonly heard, const is a contract. It's important to understand exactly what const should communicate at different points in our programs.

Embedded Job Postings

Cisco Meraki is hiring for a Firmware Features Software Engineer in San Francisco. The features engineer will architect and implement features across the software stack, spanning from device drivers and routing code all the way up to Cisco's distributed, cloud-hosted backend.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We've added code syntax highlighting to our website. Please let us know if you notice any problems with code samples after the update.

We expanded the Glossary with additional terms.

We updated the following article with new content:

New Articles

We published the following article in January:

These were our most popular articles in January:

  1. Circular Buffers in C/C++
  2. std::string vs C-strings
  3. Jenkins: Configuring a Linux Slave Node
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. Jenkins: Running Steps as sudo
  8. Demystifying Microcontroller GPIO Settings
  9. An Overview of C++ STL Containers
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

January 2019

Happy New Year!

Welcome to the January 2019 edition of the Embedded Artistry Newsletter. This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Highlights from 2018
  • The Embedded Template Library
  • A code "self-review" exercise
  • Embedded news from around the web

2018 in Review

We covered a lot of ground with our newsletter in 2018. We focused on four major topics:

  1. Improving Quality
    1. Lightweight Processes to Improve Quality
    2. Testing Embedded Systems
  2. Libraries suitable for embedded systems
    1. C and C++ Libraries We Like
    2. C++ Embedded Frameworks
    3. Three more useful embedded projects
  3. Changes to C++ that might impact embedded developers
    1. C++ standardization meeting in Rapperswil
    2. New Freestanding C++ proposal
    3. An update on Freestanding C++ & scoped enumerations proposal
  4. Security Flaws & Hacks
    1. Spectre & Meltdown
    2. Spectre Returns
    3. The Big Hack

2018 was a productive year for Embedded Artistry:

On a fun note: according to our Twitter analytics data, 99% of our followers love “dogs”, with “tech news” and “space & astronomy” following closely behind. Our team feels the same!

Embedded Template Library

We've been hard at work on our C++ embedded systems framework. A critical design requirement is that the framework core components do not utilize dynamic memory allocation. Unfortunately, this excludes most of the STL containers, as they get storage from the heap by default.

Rather than reinventing the wheel, we spent some time searching for existing solutions to this problem. We feel lucky to have discovered the Embedded Template Library (ETL), an open-source project created by John Wellbelove at Aster Consulting Ltd.

The ETL provides fixed-size alternatives to STL container types. The ETL containers specify a maximum capacity at compile-time and make no calls to malloc/free/new/delete. As an added bonus, the containers are all cache friendly: storage for all container types is allocated in contiguous blocks. The ETL also provides additional utilities and frameworks that are useful for embedded projects. The library can be safely used in conjunction with the STL.

The ETL will be included in our embedded framework and will be seen in our code samples for future articles. You can find additional ETL links and information on our website.

Code Self-Review Exercise

Code reviews are one of the essential practices for improving software quality. Aside from identifying issues with code before its deployed, reviews expose developers to new techniques, approaches, and patterns of thought.

As a consultant, it’s hard to enlist help for internal code reviews. As such, I’ve been testing other strategies to improve the code I write. For example, I use automated processes to check for problems or style inconsistencies, and I run through a checklist of common design mistakes before I merge in any new code changes.

Recently, I’ve been noting mistakes I’m prone to make and screening for them before submitting pull requests. I learned about this exercise in "Social Effects of Peer Review.". Here's a summary of the exercise from the article:

Try this experiment sometime: For one week, make a log of every error you make. Every misspelling in e-mail, every time you accidentally close the application you’re working in, every time you have a bug in your code, even every syntax error you make. Don’t get too specific; keep each item in the list high-level and use hash marks to count how many times you make that type of mistake.

For now, just imagine you’ve done this. As you might guess, certain patterns emerge. A few of the items have an awful lot of hash marks next to them. In fact, by the end of the second day you might have been getting tired of counting some of them!

And that annoyance might live in the back of your mind. You might even start thinking about it consciously. Pretty soon you’ll anticipate the mistake and prevent yourself from making it. Some of those will come easy, others will take work to eliminate. But eventually you develop habits that prevent that type of error completely. If you frequently forget to close parenthesis in your code, perhaps you’ll enable the feature in your code editor that closes parenthesis for you. If you close applications by accident, maybe you’ll pause just a few seconds every time you’re about to hit the close button. Maybe you’ll look up how to spell common words or learn how to input your common mistakes into the auto-correct facility in your e-mail editor. Over time you become more productive and more efficient. You’re working faster and smarter. Not through some self-help seven-step program, just by observing yourself a little more carefully and systematically than you’re used to.

Noting down all of the mistakes you make can also be a humbling experience. Give it a try this week as a New Year’s resolution and see how much you improve by just paying more attention to yourself.

Around the Web

MQTT and CoAP are ubiquitous in IoT applications. Unfortunately, Trend Micro reported that these messaging protocols are insecure. You can read their full report, "The Fragility of Industrial IoT's Data Backbone".

While we're on the topic of vulnerabilities, this article by The Register discusses the software bugs and software quality.

All About Circuits shared an embedded project that we love: How to Make Your Umbrella into a Weather Station with a Particle Photon.

All About Circuits also posted an introduction to MEMS systems. Since many components in our embedded systems utilize MEMS technology, we recommend familiarizing yourself with how it works.

This article provides a nice summary of how different industries are leveraging IoT devices.

IEEE Spectrum published an article describing sensors that are suitable for medical applications.

Orbit Fab launched an experiment to the ISS to test cube satellite refueling technology.

Ellisys has created a video series that teaches developers about Bluetooth. Videos include: Intro to Bluetooth Low Energy, Generic Access Profile (GAP), Advertisements, Connections, Generic Attribute Profile (GATT), Power Consumption, Security, Bluetooth 5 and IoT, and Intro to Bluetooth Mesh.

Jee Labs published a six-part series which introduces developers to the CAN bus:

  1. Intro
  2. Access
  3. STM32
  4. JeeH API
  5. Demo
  6. Single-wire

Website Updates

We updated the Embedded Artistry Technology Radar to reflect the changes in various technologies and techniques that are under assessment.

We also updated the following previously published articles with new and corrected content:

New Articles

We published the following articles in December:

These were our most popular articles in December:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. An Overview of C++ STL Containers
  8. Jenkins: Running Steps as sudo
  9. Migrating from C to C++: NULL vs nullptr
  10. Implementing Malloc: First-fit Free List

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

We’d also love to hear how the code review experiment goes for you.

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

December 2018

Welcome to the December 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Update on Freestanding C++
  • A new C++ proposal: Extensions of Scoped Enumerations
  • Code cleanup sprints
  • Bloomberg's "Big Hack" follow-up
  • Embedded news from around the web

Update on Freestanding C++

Ben Craig has been heading up the Freestanding C++ effort, which we discussed in the June 2018 and August 2018 editions of the newsletter. We are excited about a Freestanding C++ subset, as we frequently write C++ code destined for embedded devices without an OS. For most embedded platforms, we have to hack the language apart to avoid problematic areas. An official subset would be a welcome change.

In November, Ben traveled to the WG21 San Diego meeting and posted a detailed update on Reddit. The Freestanding C++ proposal continues to be discussed and refined, and WG21 San Diego included a discussion around how Freestanding C++ might work within the scope of C++ modules. Ben also reports that companies are beginning to adopt the new terminology proposed in his Freestanding proposal.

These are the latest Freestanding-related papers:

Extensions of Scoped Enumerations Proposal

We’ve been working on our framework over the past few months and have found ourselves frequently wishing for the ability to extend enumerations. We’d like to provide baseline definitions in our framework, but allow users to extend those definitions according to their specific needs. Sadly, this is not possible with C++. There are multiple proposed workarounds, but we haven’t liked anything we’ve tried thus far.

A new proposal popped up in the SG14 discussion group which recently caught our eye: Extensions of Scoped Enumerations. This paper describes improvements to scoped enumerations (enum class), which includes the ability to extend an enum class definition through an inheritance-like mechanism. The proposal's author also released a source repository which has forked clang to demonstrate the changes.

This proposal is still in the formative stages, but we are looking forward to developments in this area.

Code Cleanup Sprints

In Timeless Laws of Software Development, Jerry Fitzpatrick talks about the need to continually revise our code. Writers never dream of submitting their first draft; my own articles and newsletters go through at least two rounds of editing. However, as developers we are rarely that diligent. Most of our code is published as a first draft and not revisited unless an issue arises that forces our hand.

After reading Timeless Laws, we instituted a monthly "code cleanup sprint" at Embedded Artistry. On the last Friday of each month, we spend the entire workday poking around our repositories, refactoring, fixing TODOs, improving comments, and making other small codenimprovements.

Every software project is a massive investment of time, energy, and knowledge. Like a garden, without proper tending, it grows out of control. The more we neglect our maintenance efforts, the more work it takes to beautify and restore order to our code base. Now we look forward to these cleanup days and the deep sense of satisfaction that comes from taking the time to improve our project’s overall quality.

We hope that your teams can also dedicate a full day or more each month for tending to your project’s code base.

Bloomberg's "Big Hack" Followup

Another month has gone by and Bloomberg still stands by their Big Hack story, which we discussed in last month's newsletter.

Likewise, the companies named in the story still deny the claims. Both Amazon and Apple have pulled ads from Bloomberg's website in response to the article, and Apple has banned Bloomberg reporters from their next event. SuperMicro told Congress that they found no spy chips during their internal investigations. They claim that the “test processes at every step are designed to alert us to any discrepancies from our base design”. This seems to be an exaggerated claim based on our experiences with electronics manufacturing.

The Washington Post, owned by Jeff Bezos, continues to fire shots at Bloomberg. The article claims that another Bloomberg reporter is performing a separate analysis to try to confirm the details of the story. Perhaps Bloomberg is close to crumbling?

On the technical front, Serve the Home describes why the Big Hack is improbable in this excellent article.

As we mentioned last month, we have to doubt the validity of the story given the lack of corroborating evidence or confirmation from any named parties.

Around the Web

The security flaws keep rolling in:

The US indicted Taiwanese foundry UMC and Chinese DRAM maker Fujian Jinhua with conspiracy to steal IP from Micron. Another article, "Knowledge Transfer, or IP Theft?", provides a deeper analysis at the situation.

Mohammad Afaneh updated his excellent introduction to Bluetooth GATT. Mohammad's website is a valuable resource for anyone working on a Bluetooth device.

IEEE Spectrum published "The Case Against Quantum Computing", which provides a sobering look at the difficulties underlying the technology.

EETimes collected links to Aspencore's Road to Mars Special Project. Two interesting articles related to embedded systems:

Steve Branam revived an excellent interview from Jack Ganssle's archives, where Jack interviews James Grenning about TDD:

Website Updates

We made the following updates to the Embedded Artistry website:

New Articles

We published the following articles in November:

  1. Musings on Supply Chain Vulnerability in Light of The Big Hack
  2. A Look at Ten Hardware Startup Blunders, Pt. 1: Process
  3. A Look at Ten Hardware Startup Blunders, Pt. 2: Schedule and Focus
  4. Missing /usr/include after updating to OSX 10.14? Try this fix
  5. A Look at Ten Hardware Startup Blunders, Pt. 3: Team

These were our most popular articles in November:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Kick off a CI Build with GitHub Push Notifications
  8. Jenkins: Running Steps as sudo
  9. Implementing Malloc: First-fit Free List
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

November 2018: The Big Hack

Welcome to the November 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The Big Hack story and its ensuing drama
  • A request for modern Embedded Linux resources
  • Embedded articles from around the web
  • Website updates

The Big Hack

At the beginning of this month, Bloomberg dropped a bomb by publishing "The Big Hack: How China Used a Tiny Chip to Infiltrate America's Top Companies". They followed the initial story with details about the software side of the attack. The Bloomberg article is a thrilling and dramatic read which has stirred up quite a bit of controversy.

If you haven't read the original story, here's a brief summary:

Between 2014-2015, Supermicro server motherboards had a small IC inserted onto the PCB, supposedly connected to the baseboard management controller (BMC). The chip allowed attackers to alter “the server’s Core OS so it could accept modifications and contact attacker-controlled computers for further instructions/code”. Supermicro subcontractors installed these chips at the behest of the People’s Liberation Army (PLA) of China. The software side of the picture involves two vectors: shipping Supermicro servers with outdated firmware that contained known security vulnerabilities, and releasing firmware updates with new vulnerabilities which would be installed after the boards were already in customers’ hands.

Unsurprisingly, the story has been vehemently denied by all parties involved:

Bloomberg was transparent about the companies' denials and published them on their website. Bloomberg then doubled down on the Big Hack by publishing a follow-up article stating that new evidence of hacked Supermicro hardware was found in a (unnamed) US telecom. Unlike the "tiny chip" on the motherboard, the reported hack involved an implant built into the Ethernet connector.

The Big Hack and the ensuing wave of denials have read like a fictional spy thriller. Following the commentary surrounding the story has been fun and enlightening:

  • Joe FitzPatrick at Securing Hardware, one of the sources named in the story, shared his thoughts around hardware implants, how he'd execute such an attack, and how compromising the BMC firmware would be a much simpler approach than implementing the hardware hack.
  • Bloomberg published the story a month after the NASDAQ delisted Supermicro.
  • Supermicro shares plunged ~50% following the first announcement.
    • 20+ people made suspicious short positions and sale orders just before the story broke, leading to calls for an SEC investigations and suspicion that this is a Supermicro hit piece.
  • Matt Levine speculated on whether securities fraud would apply if companies falsely denied the story.
  • Risky Business published a podcast digging into the story and interviewed Joe FitzPatrick, who said he feels “uneasy” about the story as published.
    • Interestingly, Vice states that Joe FitzPatrick "told Motherboard in an online chat he was 'not surprised to see an example finally.'" This is an odd statement for someone claiming the story made them feel uneasy.
  • Serve the Home analyzed the story and the BMC theory.
  • Vice dug into supply chain vulnerabilities and included this interesting quote:
    • “There are two possible stories here,” Matthew Green, associate professor at Johns Hopkins University, tweeted about the attack. “One is that there was an attack. The other is that a large swath of the National Security establishment is promoting the idea that there was an attack. Pick your poison.”
  • Sup China takes a similar tack as Matthew Green and postulates that government sources may be trying to generate or amply mistrust of China.
  • One Bloomberg opinion piece relates the hack to the growing conflicts between the US and China. Another opinion piece postulates on how hopelessly vulnerable global supply chains are - further stirring the pot in regards to China's business practices.

Now that a month has passed, we seem to be at an impasse. The lack of evidence of hacked hardware, a list of affected SKUs, or confirmation by other agencies casts doubt on Bloomberg’s story. The companies named continue to deny the story, and Bloomberg will not make a retraction, name sources, or provide further details. Other news agencies have been unable to independently confirm the story.

Regardless of whether the story is true or false, it highlights the vulnerability of our modern technological supply chains. I published my musings on modern supply chains and the feasibility of hardware tampering on the website.

A Request for Modern Embedded Linux Resources

I've received emails asking for recommendations for up-to-date Embedded Linux resources. We primarily focus on writing software for microcontrollers, so I have little experience with Embedded Linux. Most of the reference material that I've seen is at least a decade old or focuses on Raspberry Pi development using python libraries.

If you're an Embedded Linux developer who writes drivers or other low-level code, I'd love to hear about your favorite books, websites, and courses. Where do you turn to when you have a problem? What training do you take to improve your skills? What tools do you use most often?

You can respond directly to this email or send us a message on Twitter.

Around the Web

Another day, another security flaw announcement, this time for Amazon FreeRTOS.

Phil Koopman highlighted the cost of producing highly safety-critical software.

In the December 2017 edition of the newsletter we covered advances in chip design techniques. Read this IEEE article for another new chip design technique: Through-Silicon Transistors Could Make Stacking Chips Smarter.

Meeting Embedded shared a short interview with Dan Saks, a champion of embedded C++ development.

Mohammad Afaneh at Novel Bits continued publishing his Bluetooth Mesh tutorial series:

Jack Ganssle finished publishing "Top 10 Reasons Embedded Projects Get Into Trouble". He finished his serious with two of the most common problems I see: "Quality Gets Lip Service" and "Unrealistic Schedules".

Website Updates

The Glossary received another extensive update focused on manufacturing and supply chain terminology.

We updated our Technology Radar to include the Espressif ESP32 chipset. Saket Vora introduced us to this chip, which features 512kB SRAM, qSPI supporting 16Mbit external flash, low deep sleep current, and a superb datasheet. We're evaluating it for use with our embedded framework.

New Articles

We published the following articles in October:

We also updated the following articles with new content:

These were our most popular articles in October:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Jenkins: Kick off a CI Build with GitHub Push Notifications
  9. Migrating from C to C++: NULL vs nullptr
  10. Implementing Malloc: First-fit Free List

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next email, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

October 2018: EA Framework Announcement & More Embedded Libraries

Welcome to the October 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Embedded Artistry's C++ Framework Project
  • Three more projects embedded developers should be aware of
  • Embedded articles from around the web
  • Website updates

Embedded Artistry Framework Project

Over the past two years in San Francisco, I’ve had the opportunity to consult with dozens of companies and to work on a wide variety of embedded products. Most teams and projects seem to struggle with the same issues, especially with hiring firmware engineers and keeping projects on schedule.

Why is everyone running into the same roadblocks?

In order to find solutions, I've been immersing myself in the software engineering corpus, reflecting on the coupling between firmware and hardware, learning software architecture, practicing Test Driven Development, and researching embedded libraries and frameworks that can help firmware developers move faster.

What I’ve noticed is that the software industry has learned a lot of lessons over the past fifty years, but we’ve failed to incorporate many of them in the embedded community. We are building products in less time, with minimal teams, and generally using poorly tested vendor SDKs. We aren’t decoupling our software from the underlying hardware. Many embedded teams still don’t utilize modern development practices such as static analysis, unit testing, and continuous integration.

What if we actually leveraged the lessons we’ve learned as an industry? What if we built our embedded software in such a way that we minimized coupling with the underlying hardware and RTOS?

  • We could begin developing and testing our embedded software before we have hardware in house
  • We could write device drivers on our host machine (talking to real parts) and port them over to our target hardware with full confidence that they will work
  • We could defend against the inevitable changes in components and processors that trigger software rewrites and massive schedule delays
  • We could leverage proven libraries and frameworks to save time and avoid reinventing any wheels
  • We could automate our testing and check our code often for preventable bugs

Embedded Artistry is building an embedded C++ framework which will help us accomplish these goals. We want to provide embedded teams with a solid foundation that incorporates our lessons learned and supports modern development practices.

Our primary goal is to support decoupling firmware from hardware at various levels of abstraction (platform, circuit board, processor). Rather than aim for universal abstractions and interfaces, we want to provide a minimal foundation which teams can expand to meet their product’s specific goals and requirements.

We are currently building our basic framework structures on a host machine to ensure we can develop software off-target. Later this fall we’ll bring the framework up on a couple of common embedded platforms.

I’ll be sharing more about the framework as it evolves over the coming months. In the mean time, I’d love to hear from you if you think your team could benefit from a system like this.

Three More Useful Embedded Projects

I always have my eyes open for interesting embedded projects. I want to share three more projects that I've discovered, two of which are being actively used in our framework:

  • Embedded Template Library
  • mpaland/printf
  • embxx

For other embedded projects, libraries, and frameworks which we've shared, check out the following editions:

Embedded Template Library

A common concern with using C++ and the STL on embedded systems is the reliance on exceptions and dynamic memory allocation. For many embedded systems, use of these features is disallowed. Working around this limitation is possible, but often more difficult than project schedules allow for.

John Wellbelove of Aster Consulting Ltd created the Embedded Template Library (ETL) to address these issues. The ETL provides fixed-capacity STL container alternatives and requires no dynamic memory allocation, no RTTI, and no exceptions. The ETL also provides additional container types, helpful utilities, algorithms (such as CRC), reusable framework components (FSM, state chart, scheduler, callback timer), and an abundance of documentation.

If your embedded compiler doesn't support modern C++ standards, you're in luck - the library is compatible with C++03, and many of the C++11 features have been backported through the ETL.

The ETL is an becoming an essential tool for our embedded C++ development. We've adopted the ETL in our current framework project to take advantage of fixed-capacity containers and other useful features.

mpaland/printf

printf is one of those commands that every developer relies on. However, using printf on many embedded systems can be a challenge due to the size of the library (20kB+ is not uncommon). Perhaps you do not want to link with libc for your platform, or the implementation you're using requires you to implement esoteric system functions for each platform.

The mapaland/printf implementation addresses these problems. The implementation is ~600 lines and has no external dependencies. The library is reentrant and all of the useful flags (including floating point and precision/width specifiers) are supported. All you have to do to get up and running on your platform is define a _putchar() function which outputs a character over your chosen communication bus (UART, USB, BLE, etc.).

We've integrated mapaland/printf into our embeddedartistry/libc and are actively using it in our framework project.

Embxx

If you're working with C++ on embedded systems, chances are that you've come across the Practical Guide to Bare Metal C++. This free e-book is an excellent starting point for applying C++ to embedded systems, and the author walks through the creation of many useful embedded systems constructs.

The concepts presented in the book are fleshed out in the companion embxx library. embxx is written in C++11 and targeted for bare metal and embedded Linux systems. The library doesn't use RTTI or exceptions, and most of the constructs presented in the library utilize static memory allocation.

Sadly, the project is no longer active, but it still serves as an excellent resource for embedded C++ developers. Many of the utilities, such as StaticFunction, StaticQueue, and StaticPoolAllocator, are always useful for embedded systems. The project also serves as a great reference for how we can utilize template meta-programming and static memory allocation schemes when writing embedded C++ software.

Around the Web

Steve Branam at Flink And Blink published "So You Want To Be An Embedded Systems Developer". This is an excellent article for those just starting out in the embedded systems field.

Matt Chernosky at Electron Vector shared 7 tips for adding unit tests to existing firmware. Start with Matt's tips if you need inspiration for adding tests to your existing project.

Mohammad Afaneh at Novel Bits published a four-part tutorial on Bluetooth Mesh. If you want to learn about Bluetooth Mesh, start with this series.

Mohammad also shared an interesting white paper in his weekly Bluetooth newsletter: Bluetooth Angle Estimation for Real-Time Locationing. The paper discusses using Bluetooth Angle of Arrival and Angle of Departure for indoor locationing.

Jack Ganssle continues to publish his "Top 10 Reasons Embedded Projects Get Into Trouble". We've worked our way to Number 3:

Website Updates

Our Glossary page has been expanded. If you notice any terms or acronyms that are missing, let us know!

We've restructured our Libraries page and added more of our own open-source projects to the list.

Our Technology Radar has been updated:

  • We are now using TDD and recommend it for adoption
  • We are recommending the Embedded Template Library for adoption after completing our trial period
  • We are experimenting with automated clang-tidy checks on our framework project
  • We have moved away from Doctest and are now trying out Catch for our C++ unit testing because it supports -fno-exceptions

New Articles

The following article was published on our website in September:

These were our most popular articles in September:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Migrating from C to C++: NULL vs nullptr
  9. Implementing an Asynchronous Dispatch Queue
  10. Jenkins: Kick off a CI Build with GitHub Push Notifications

Thanks for Reading!

September 2018: Testing Embedded Systems

Welcome to the September 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Testing Embedded Systems
  • Embedded articles from around the web
  • Embedded Artistry website updates

Testing Embedded Systems

Embedded Artistry was founded with the goal of creating reliable, safe, and well-tested embedded systems. The sad fact is that most embedded software that we've encountered is low-quality and untested. Perhaps this holds true for most of the software industry - the continued procession of hacks, flaws, and errors is discouraging.

We can talk about why this is all day long, instead we want to focus on ways that we can all improve at our craft.

We'll cover three facets of testing embedded systems:

  1. Unit testing
  2. Debugger-based testing using metal.test
  3. Phil Koopman's lectures on embedded software quality and testing

This is not an exhaustive list, and there is much more to say about software quality that we’ll continue to cover in future articles and newsletters. For today, focusing on testing is a great place to start.

Unit Testing

Most developers know they should be writing unit tests as they develop new features. If you're unfamiliar with the concept, unit testing focuses on testing the finest granularity of our software, the functions and modules, in an independent and automated manner. We want to ensure that the small pieces operate correctly before we combine them into larger cooperating modules. By testing at the finest granularity, we reduce the total number of test combinations that are needed to cover all possible logic states. By testing in an automated manner, we can ensure that any changes we make don't introduce unintended errors.

"A key advantage of well tested code is the ability to perform random acts of kindness to it. Tending to your code like a garden. Small improvements add up and compound. Without tests, it's hard to be confident in even seemingly inconsequential changes." -Antonio Cangiano (@acangiano)

Unfortunately, at Embedded Artistry we’ve only worked on a handful of projects that perform unit testing.

The primary reason for this is that many developers simply don’t know where to start when it comes to writing unit tests for embedded systems. The task feels so daunting and the schedule pressures are so strong that they tend to avoid unit testing all together.

After receiving a request for more resources on Test Driven Design (TDD) with embedded systems, we decided to share our favorite unit testing & TDD resources.

James Grenning and TDD

James Grenning has put a tremendous amount of effort into teaching embedded systems developers how to adopt TDD. He published an embedded systems classic, Test-Driven Development for Embedded C, and regularly conducts TDD training seminars.

James has written extensively about TDD on his blog. Here are some of our favorite posts:

You can also watch these talks for an introduction into the how and why of TDD:

Phillip is signed up to take James Grenning's TDD webinar, which begins today. If you're interested in taking a training class, look into these links:

Matt Chernosky and Electron Vector

Matt Chernosky, who runs the Electron Vector blog, is an invaluable resource for embedded unit testing and TDD. If you are having a hard time getting started with unit testing and TDD, Matt's articles provide a straightforward and accessible approach.

Here are some of our favorite articles from Matt's blog:

Matt published a free guide for using Ceedling for TDD. If you are experienced with TDD but are looking to improve your Ceedling skills, check out his Ceedling Field Manual.

Throw the Switch

Throw the Switch created the Ceedling, Unity, and Cmock unit testing suite. This is the trio of tools that Matt Chernosky uses and writes about.

Aside from creating testing tools, Throw the Switch maintains a library of test-related articles and a course called Unit Testing & Other Embedded Software Catalysts. Additional courses related to unit testing for embedded systems are being developed.

Unit Testing Frameworks

Listed below are frequently recommended unit testing frameworks for C and C++. There are more unit testing frameworks in existence than we can ever review, so again this list is not exhaustive. If nothing jumps out at you in this list, keep looking to find one that fits your team’s development style.

We previously covered the Throw the Switch C unit testing frameworks: Unity, Cmock, and Ceedling.

Cmocka is the C unit testing framework we started with. Cmocka ships with built-in mock object support and operates similarly to Unity & CMock. The framework is built with C standard library functions and works well for embedded systems testing.

Catch appears to be the most popular C++ unit testing framework. Catch is a header-only library which supports C++11, C++14, and C++17.

Doctest is the unit test framework we use for EA’s C++ embedded framework project. Doctest is similar to Catch and is also header-only. Our favorite attribute of Doctest is that it keeps the test code alongside the implementation code. Doctest also enables you to write tests in headers, which Catch does not support.

GoogleTest is Google's C++ unit testing framework. GoogleTest is one of the few C++ frameworks with built-in mocking support.

CppUTest is a C++ test suite that was designed with embedded developers in mind. This framework is featured in James Grenning's book Test-Driven Development for Embedded C. C++ features within the framework are kept to a minimum enabling it to be used for both C and C++ unit testing.

If you're interested in mock object support for C++, check out GoogleMock, Trompeloeil, and FakeIt. Each of these mocking frameworks can be integrated with the unit test frameworks mentioned above.

Embedded Debugger-Based Testing

We always want to run as many tests as possible on a host PC when unit testing embedded systems code. However, we can’t test every aspect of our system on a host machine. Before shipping the final system, we need to evaluate the target compiler, issues which only present themselves on the target (e.g. endianness, timing), and the actual target hardware and hardware interfaces.

Metal.test is a framework which can help us with on-target testing. This project is maintained by Klemens Morgenstern, an independent contractor and consultant. Metal.test enables automated execution of remote code using debugger hardware, such as J-LINK or ST-Link. The project currently supports gdb, and lldb support is planned for a future release.

Metal.test features:

  • I/O Forwarding
  • Code Coverage
  • Unit testing
  • Call tracing
  • Profiling
  • Function Stubbing at link-time
  • Extensive documentation (in the GitHub repository Wiki)

Metal.test also includes a plugin system. While it's not essential, plugin support enables developers to extend the functionality to support any use case that a debugger can support.

Klemens is looking for feedback on metal.test. Don't hesitate to reach out with questions, issues, or other feedback.

Phil Koopman on Testing & Software Quality

Phil Koopman is a professor at Carnegie Mellon University. He also runs the Better Embedded Software blog, which is a must-read for embedded developers.

Phil has produced an immense and invaluable body of work, much of it focused on embedded software quality. The lecture notes for his embedded systems courses are available online, and he regularly posts lecture videos on his Youtube channel.

Here's a selection of his lectures that are related to testing and software quality:

Around the Web

Mohammad Afaneh has finished his Introduction to Bluetooth Low Energy eBook. If you're working with Bluetooth and BLE, Mohammad is the person to follow.

NAND makers are striving for larger storage capacities. There are announcements of 512 Gbit, 1 Tbit, 1.33 Tbit capacities. SK Hynix is also working on a package with a whopping 8 Tbit capacity.

Jack Ganssle has started a blog at the end of July. Jack is also publishing a weekly series on embedded.com which expand upon his "Top 10 Reasons Embedded Projects Get Into Trouble". So far, he's worked his way up to number 7:

You should also check out Phil Koopman’s slides for Avoiding the Top 43 Embedded Software Risks. How many risks has your team taken on?

Website Updates

We've added a list of our favorite software-related books to the Support page. If you're looking for a new programming or architecture book, you can support the website and newsletter by using the Amazon links found there.

Our popular Circular Buffers in C/C++ article was re-written based on learnings and feedback received over the past year.

The Embedded Artistry Technology Radar has received its first update with new techniques, libraries, and tools.

Three of our Resource pages have a brand-new look:

Our Glossary is always expanding and has been updated with new terms.

New Articles

These articles were published on our website in August:

These were our most popular articles in August:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Jenkins: Running Steps as sudo
  7. C++ Casting, or: "Oh No, They Broke Malloc!"
  8. Implementing an Asynchronous Dispatch Queue
  9. Migrating from C to C++: NULL vs nullptr
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

August 2018: Spectre Returns, Freestanding C++, Technology Radar

Welcome to the August 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • More Speculative Execution attacks: SpectreRSB and NetSpectre
  • The new version of the Freestanding C++ proposal
  • The Embedded Artistry Technology Radar
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Speculative Execution Attacks

Earlier this year we covered Spectre and Meltdown, two speculative execution vulnerabilities that affect a majority of the world's processors. In June, there was another announcement regarding an Intel speculative execution vulnerability dubbed lazy floating-point state restore.

The trend continues, and two more speculative execution attack vectors have come to light in the past few weeks:

  • SpectreRSB, which exploits speculative execution of the return stack buffer
  • NetSpectre, a remote attack which does not require attacker-controlled code to run on the victim's machine

Exploring these speculative execution vulnerabilities is a great way for embedded developers to learn the details about how processors work.

SpectreRSB

SpectreRSB is a speculative execution vulnerability which exploits the return stack buffer (RSB). The RSB is a structure similar to the branch predictor which is used to predict return addresses. When a call instruction is issued, the RSB pushes the return address onto an internal hardware stack.

SpectreRSB has six variants, but the basic flow is as follows:

  1. Context switch to the attacker:
    • The attacker flushes shared address entries (i.e. flush reload)
    • The attacker pollutes the RSB with the target address of a payload gadget in the victim’s address space
  2. The attacker yields the CPU back to the victim
  3. The victim eventually executes a return
    • Speculative execution occurs at the address which the attacker injected into RSB
  4. Control switches back to the attacker, data is leaked

The authors of the paper claim that the existing mitigations against Spectre variant 1 do not protect against SpectreRSB. Intel claims that SpectreRSB is related to branch target injections which have already been mitigated. On some processors, an RSB refilling patch is available that interferes with SpectreRSB’s attack mechanism.

Intel, AMD, and ARM processors are affected, as they all utilize RSBs to predict return addresses.

More on SpectreRSB:

NetSpectre

The NetSpectre vulnerability is related to Spectre variant 1 in that it uses speculative execution to perform bounds-check bypass. NetSpectre can be used to defeat address-space layout randomization on a remote system.

Unlike other Spectre variants, NetSpectre does not require a victim to download and run malicious code their machine. Instead NetSpectre only requires the victim to have an active network connection. NetSpectre variations are currently too slow to be valuable to attackers, with exfiltration speeds between 15 and 60 bits/hr. Even if the attack vector is not currently usable, it demonstrates that Spectre is not simply a local vulnerability.

It’s reported that all CPUs affected by the Spectre variant 1 are also affected by NetSpectre, therefore existing vendor mitigations should be enough to protect against it. These mitigations require firmware and OS updates, so unpatched devices remain vulnerable.

More on NetSpectre:

New Freestanding C++ Proposal

In the June 2018 Newsletter, we mentioned the C++ standards proposal to support a "Freestanding" C++ language subset which is suitable for use without an underlying operating system.

A new version of the Freestanding proposal has been released: P1105R0: Leaving no room for a lower-level language: A C++ Subset.

The primary goal of the proposal is to make core language features (such as exceptions, RTTI, and thread local storage) optional if they have an OS dependency or incur storage space overhead. This would better-enable C++ to support the embedded systems and kernel programming communities.

For more on Freestanding C++:

Embedded Artistry's Technology Radar

I was introduced to the Technology Radar concept while at Bredemeyer Consulting's Software Architecture Workshop. The Technology Radar concept was created by ThoughtWorks, and they regularly publish updates to their own radar.

I immediately latched onto the concept as a way to keep track of the technologies and techniques that we are investigating at Embedded Artistry. The technology radar also helps us document the solutions that we recommend others adopt.

We’ve identified four categories that we plan to explore as an embedded systems consulting firm:

  1. Tools
  2. Libraries and Frameworks
  3. Platforms (both hardware and RTOS)
  4. Techniques

We've released the first-draft of our technology radar and will update it on an ongoing basis.

You can find our Technology Radar on the website under the Resources menu.

For more on technology radars:

Around the Web

IT Hare has published another article on (Re)Actors: CAS (Re)Actor for Non-Blocking Multithreaded Primitives.

An amusing RISC-V smear campaign by ARM has brought a lot of public attention to the RISC-V architecture. If you're unfamiliar with RISC-V, check out this IEEE article: RISC-V's Open-Source Architecture Shakes Up Chip Design. You can learn more on the RISC-V Foundation website.

Segger, who produces the J-Link and J-Trace tools, has announced their own C-based embedded OS called emPack. In addition to an operating system, Segger is also providing emPack plugins for GUI, File System, TCP/IP, USB-Device, USB-Host, and more.

Website Updates

We've launched a Store page, where we will be adding templates and informational products that embedded teams can leverage. Both free downloads and paid downloads are in the Store.

We've created a Support page which will stay updated with the ways you can support the website.

We updated the following blog posts this month:

Our Glossary is always expanding and has been updated with a plethora of new terms.

As mentioned we added a link to the Embedded Artistry Technology Radar under the Resources menu.

New Articles

These articles were published on our website in July:

These were our most popular articles in July:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. Jenkins: Running Steps as sudo
  6. An Overview of C++ STL Containers
  7. Implementing an Asynchronous Dispatch Queue
  8. A Simple Consulting Services Agreement
  9. Creating and Enforcing a Code Formatting Standard with clang-format
  10. Demystifying ARM Floating-Point Compiler Options

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

July 2018: Anniversary Edition

Welcome to the July 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month marks the one-year anniversary of our newsletter! In this edition, we’ll cover:

  • Looking back at major developments over the past twelve months
  • A new Intel vulnerability announcement
  • C++ conference talks with an embedded focus
  • The new I3C specification
  • Maxim's new DARWIN MCU line
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Looking Back

We’ve been publishing our newsletter for one year. The past twelve months have been eventful for embedded systems. Here are some of the events that stood out to us:

Intel Vulnerability Announcement

A new Intel vulnerability has been announced, dubbed "lazy floating-point (FP) state restore". This vulnerability affects all Intel Core-based microprocessors.

Intel provides a lazy floating-point state restore technique, which delays restoration of a processing state until an instruction operating on that state is executed by a new process. An attacker can take advantage of this technique to infer register values of a different process through a speculative execution side channel. This can be used to retrieve AES encryption keys, which are often stored in the floating-point unit (FPU) registers.

Intel recommends that developers use Eager FP state restore to prevent this vulnerability.

For more on lazy FP state restore:

Embedded C++ Talks

A lot of great Embedded C++ conference talks have been published over the past few months. Here are some of our favorites:

The I3C Specification

A typo led me to discover that the MIPI Alliance has announced the I3C Specification, the successor to I2C. I3C also intends to replace the SPI bus, which has no clearly defined standard and sports multiple implementations. Since GPIO availability is always contentious, many board designers will gladly switch to I3C's two-wire bus if it lives up to the hype. SPI will still maintain an advantage over I3C in high-speed point-to-point use cases, like SPI flash devices.

I3C is still a two-wire interface but unlike I2C, I3C will provide a minimum data rate of 10Mbps. I3C will also support multi-master, slave-initiated communication, dynamic addressing, and power management features (eg. sleep mode). A broadcast address is provided, allowing masters to send command codes to all slave devices at once. I3C lines will be push-pull instead of open-drain, enabling better power consumption per transferred bit. Most I2C slave devices will be able to operate on an I3C bus as long as they have a 50ns glitch filter and do not stall the clock.

I3C is brand new, so I have yet to find any sensors or processors that have shipped with I3C support. I expect we’ll start to see some devices announced later this year and more announced next year.

For more on I3C:

Maxim's DARWIN MCUs Line

Maxim has historically been in the power-IC business, focusing on components such as regulators, fuel gauges, and PMICs. They recently announced a low-power processing line: DARWIN.

The DARWIN MCU line is split into two families:

  • Generation U: Ultra-low power, with flash sizes from 256KB to 2MB, and SRAM sizes ranging from 96KB to 512KB
  • Generation UP: Ultra-low-power + performance, with 3MB flash and 1MB SRAM

The Generation U MCUs feature a Cortex-M4 with an FPU and can be clocked up to 96MHz. Maxim is capitalizing on their power management specialty with their MCUs. The processors include an embedded PMU, provide flexible power modes, and support dynamic clocking (down to 4MHz on the processor clock).

The Generation UP MCU is also a Cortex-M4. The UP MCU can be clocked up to 120MHz and includes SPI execute-in-place (XIP) interfaces. A memory decryption integrity unit (MDIU) provides on-the-fly decryption of data stored in external flash. Power management capabilities are further extended by providing power control over individual SRAM banks.

The Darwin MCU line looks to be an interesting contender in the low-power microcontroller space.

For more on the DARWIN MCUs:

Around the Web

ITHare published A Useable C++ Dialect That is Safe Against Memory Corruption. This article covers using (Re)Actors to help ensure memory safety in our programs.

All About Circuits takes a look at IP tension in Who Really Owns Hardware: Property Rights vs Copyrights

Another great All About Circuits article was Do Engineers Design Anything Anymore? Electrical Engineering in the Age of ICs.

Atomera plans to breathe new life into older chip manufacturing with a technique that involves burying atom-thin layers of oxygen below the surface of a transistor's silicon. This technique reduces variability among transistors and enables designers to make use of larger process technologies.

The Wi-Fi Alliance has announced the Enhanced Open certification to provide improved privacy protections and security on open Wi-Fi networks.

Website Updates

We have done some major revamping of our website this month!

New Articles

The following articles were published on our website in June:

These were our most popular articles in June:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. Installing LLVM/Clang on OSX
  4. An Overview of C++ STL Containers
  5. std::string vs C-strings
  6. Jenkins: Running Steps as sudo
  7. A Simple Consulting Services Agreement
  8. Implementing an Asynchronous Dispatch Queue
  9. C++ Casting, or: Oh No, They Broke Malloc!"
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!

-Phillip

June 2018: C++ Standard Proposals for Rapperswil Meeting

Welcome to the June 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover 5 C++ language proposals that will impact embedded developers.

C++ Standardization Meeting in Rapperswil

Today marks the start of the C++ standardization committee meeting in Rapperswil, Switzerland. Developers from around the world will meet and work on the future of the C++ language.

C++ is complex, widely adopted, and must meet the needs of developers of all varieties. There are always competing priorities and needs, and it is up to the standardization committee (and an array of special interest groups) to ensure the proper balance is met.

I want to share five C++ proposals which are of interest to embedded systems developers. It may be years before we see these proposals in a formal C++ standard (if they are adopted at all), and the proposals should be expected to evolve with feedback from the standardization committee.

Even though they may be years out, these ideas can still benefit you today. Many of the proposals have corresponding testing libraries enabling you to try out the new features in your own programs. It's also exciting to take a look under the hood and see what future language changes are being discussed.

The proposals we will review are:

  1. P0709R0: Zero-overhead deterministic exceptions
  2. P0829R2: Freestanding
  3. P1028R0: status_code and standard error object
  4. P0037R5: Fixed-point real numbers
  5. P0132R1: Non-throwing container operations

For more on the Rapperswil meeting:

P0709R0: Zero-overhead Deterministic Exceptions

The Zero-overhead Deterministic Exceptions proposal is a potential game-changer for the C++ language.

The opening of Herb Sutter's proposal states that over 50% of C++ developers report that exceptions are banned in part or all of their code. As an embedded developer, I fall into both the partial- and complete-ban categories on my systems. Since exceptions are required to use many core C++ language features (e.g. constructors), this means that a large number of C++ programs are written in non-standard C++ dialects.

This proposal aims for a re-conception of the C++ exception model with two primary goals:

  1. Remove all technical reasons for a C++ project to disable exception handling
  2. Facilitate re-unification of divergent error reporting styles

The primary method of achieving the two goals is to annotate functions with the throws keyword, indicating that the function throws a statically specified type by value. Exception handling implementations can then be as efficient and deterministic as a local return-by-value, incurring no dynamic or non-local memory overhead.

How can this be accomplished? By ensuring that the only type of exception that can be returned is a std::error exception (discussed below in P1028). If the function fails, it will throw the std::error object as-if returning an alternate return value on the stack. Two additional possibilities are returning errors in unused processor registers and repurposing unused CPU flag bits to indicate whether the registers contain a return value or error.

The new exception mechanism provides two return paths for a function: one for the success case, and one for the failure case, where failure returns an error code as usual. The resulting API designs will be greatly simplified: no longer will error codes monopolize our return value channels, and users can't forget to check errors.

Those interested in trying out the new exception model are in luck. Niall Douglas has published the Outcome 2.0 library, which is effectively a library-only implementation of Herb's exception proposal.

For more on Zero-Overhead Deterministic Exceptions:

P0829R2: Freestanding

The Freestanding C++ proposal is one of the most interesting for the embedded world. Ben Craig is interested in identifying a subset of the C++ language which is suitable for kernel and embedded development.

The core goal is to enable developers to easily identify features which can be implemented without an OS. This is currently impossible with the existing standard, and many teams simply maintain a manual list of features which cannot be used in an embedded environment.

The paper identifies a working list of functions from existing headers (such as <cerrno>, <cstring>, and <algorithm>) which are allowed in a freestanding environment. The paper also identifies new headers which will need to be added and clearly outlines featuers which are not being considered for the freestanding subset.

Even if a Freestanding C++ is never formalized, the paper serves as a starting-point for features that are safe to use in an embedded environment. I will update the Embedded Artistry libc to incorporate some of the freestanding proposal notes.

For more on Freestanding C++:

P1028R0: status_code and standard error object

The standard error object proposal is a key component of Zero-Overhead Deterministic Exceptions. Even better, this proposal is compatible with the Freestanding C++ proposal, so all of the features can be used standalone on embedded systems.

This proposal covers the replacement of the <system_error> header with a lighter-weight and portable design. We also get a new std::error type with the following properties:

  • Always indicates a failure (no more "0" success codes)
  • No default construction
  • No empty state possible
  • Trivially copiable
  • Small enough to return in CPU registers (current proposal uses two pointers)
  • Portable across domains & operating systems

All operations which can be constexpr have been updated to be so, and the rest have been made literal or trivial to the maximum extent possible. Header-only libraries will now be able to safely define custom error code categories (you'll be unpleasantly surprised if you try this today). We can finally drop the if(!error_code) model, opting instead for a clearer if(ec.success()) or if(ec.failure()) check.

If you are interested in trying out the proposed status_code and std::error changes, you can try out Niall's status_code library. The Outcome 2.0 library, which demonstrates the exception proposal, also integrates the status_code support.

For more on the error code overhaul:

P0037R5: Fixed-Point Real Numbers

Fixed-point numbers are common in embedded systems for a variety of reasons:

  • Some systems don't have floating point support
  • Floating point support is often not thread-safe unless properly protected
  • Many external components communicate using a fixed-point format
  • The system may need to convert a number to fixed-point before sending the value out over a communication bus

Because there is no standard option available, developers are required to spin their own implementation or import one of the many fixed-point mathematics libraries.

Rather than treating fixed-point numbers as a special case, this P0037 proposes a drop-in integer replacement with fixed-point support. After all, an int is simply a fixed-point number with an exponent of 0. An API with the necessary fixed-point arithmetical tools will be provided by the standard.

John McFarlane's Compositional Numeric Library, which serves as a proof-of-concept for both P0037 and P0554, contains an in-development implementation of the proposed fixed_point class template. If you need fixed-point operations in your next project, give it a try.

For more on Fixed-Point Real Numbers:

P0132R1: Non-throwing container operations

As we covered earlier, many C++ projects operate with exceptions disabled. This is a problem since many of the standard library features are built around exceptions, especially allocators and constructors. If exceptions are disabled and an allocation fails, your program will typically end up in the abort function, which prevents any chance of error recovery.

P0132 proposes to create container operations which will not throw an exception on failure. The proposed additions to the standard are:

  • Allow an allocator to fail without an exception
    • Add a separate function for non-throwing allocation which returns nullptr on failure
  • Add a non-throwing reserve operation to vector and string
    • This would utilize the support for allowing an allocator to fail without an exception
  • Add a non-failing push_back to vector and string
    • Pushing without sufficient capacity will be undefined behavior
  • Add a push_back that can fail to vector and string
    • This function will report an allocation error, but will do so without an exception
  • Add a push_back that can fail to vector and string
    • This function will have no effects if allocation fails, and will not throw

As an embedded C++ developer, I would love to see these features and look forward to improved container operations in the coming years. I haven't yet seen a library to test such support, which means you will need to modify your local libc++ implementation to try these features out.

For more on Non-throwing Container Operations:

Around the Web

Watch John Regehr's closing keynote from C++Now 2018: Undefined Behavior and Compiler Optimizations.

GitNStats is a tool for analyzing how often particular files are modified. By identifying files with a lot of churn, you identify potential problem areas in your code base.

Jacob Benigo postulates on the soon-to-be-extinct embedded software engineer.

Sensirion recently announced the SDP3x differential pressure sensor family, which they claim is the world's smallest differential flow sensor.

Microsoft has announced Vcpkg, a C++ library manager for Linux, macOS, and Windows.

Website Updates

The For Beginners page has been reorganized, and a table of contents has been added.

The Hardware References page now has a table of contents. I also added Circuit Cellar's compendium of quick prototyping solutions.

The Software References page has been reorganized and a table of contents has been added. Paul Watt's blog was added to the software blog list.

New Articles

These articles were published on our website in May:

These were our most popular articles in May:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. Jenkins: Configuring a Linux Slave Node
  4. std::string vs C-strings
  5. Jenkins: Running Steps as sudo
  6. A Simple Consulting Services Agreement
  7. An Overview of C++ STL Containers
  8. C++ Casting, or: Oh No, They Broke Malloc!"
  9. Implementing an Asynchronous Dispatch Queue
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!

-Phillip

May 2018: C++ Embedded Frameworks

Welcome to the May 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Four open-source C++ embedded frameworks that can jump-start your next project:
    • Kvasir
    • modm
    • crect
    • EMB2
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Open-Source Embedded Frameworks

In last month's newsletter we recommended quality C and C++ libraries to give your next project a boost.

This month we are taking things a step further by diving into embedded C++ frameworks which can be used as the foundation for your next product or project. While each framework is focused on solving a distinct problem, there is at least one that will benefit your team.

We will cover the following four embedded frameworks:

  1. Kvasir, which specializes in compile-time checking of Special Function Register usage
  2. modm, which utilizes vendor-provided data to generate drivers and BSPs
  3. crect, which generates a deadlock-free and data-race-free scheduler for real-time systems at compile time
  4. EMB2, which provides a paradigm for developing multi-core and mutli-chip embedded programs

Kvasir

Kvasir is a C++11 framework created by Odin Holmes. Kvasir is suitable for tiny bare-metal applications such as ARM cores. By using C++11’s constexpr feature and template meta-programming, Kvasir’s features come at no cost. In some cases, the meta-programming magic provides superior implementations to common hand-coded approaches!

The primary feature is the Kvasir::Register abstraction, which enables strongly typed register and bit-field definitions and provides SFR read/write methods. These abstractions improve type safety and enable static checking of register interactions during compilation. As long as the register definitions are correct, Kvasir will eliminate off-by-one errors, prevent users from writing to reserved bits, ensure that bit-field accesses don't overflow into other fields, and generate warnings if you are incorrectly using a register that performs a clear-on-read operation.

In addition to SFR abstractions, Kvasir also supplies the Kvasir::IO abstraction to provide similar benefits with GPIO. The IO abstraction utilizes a pinLocation concept to define GPIOs and provides constexpr meta functions such as makeOpenDrain(), makeOutput(), set(), clear(), and read().

SFR bugs are extremely common on embedded systems. Eliminate them altogether by building your next system with Kvasir.

For more on Kvasir:

modm

modm (Modular Object-oriented Development for Microcontrollers) is a C++14 framework built by Niklas Hauser and Fabian Grief. The modm project uses vendor-provided chip data with a library builder, enabling modm to automatically generate startup code, chip-specific drivers, external drivers, and BSPs for your platform. Since modm provides a portable HAL, you can easily migrate your software from one processor to another supported processor with minimal effort.

modm provides a framework which is suitable for bare-metal systems ranging from the ATtiny to a 32-bit ARM Cortex-M. The HAL features no memory allocations, low RAM consumption, and lightweight stack-less threads. The framework also provides useful algorithms suitable for bare-metal systems, as well as drivers for a wide variety of SPI and I2C peripherals. modm is well-tested, featuring 64 test groups with 343 test cases and over 4000 assertions. While the HAL is not fully tested in an automated manner, a variety of example hardware projects are regularly checked by the CI server.

The modm framework currently supports ~1350 AVR and ARM Cortex-M microcontrollers from Atmel, ST, and NXP. If you are using a processor from of those vendors, modm can provide your team with a stable foundation of drivers and the advantage of being able to quickly migrate your software to another processor.

For more on modm:

crect

The crect project (pronounced correct) is a C++14 framework for generating a scheduler for Cortex-M microcontrollers at compile-time. crect uses the Cortex-M's Nested Vector Interrupt Controller (NVIC) to implement a Stack Resource Policy (SRP) scheduler which guarantees deadlock-free and data-race-free execution.

crect is built upon the Kvasir Meta-programming Library, which is also the foundation of the Kvasir framework. The use of C++ meta-programming and C++14 features allows priority ceilings and interrupt masks to be calculated at compile time. Resource locks are handled through RAII, and resource access is handled using a monitor pattern).

The most impressive thing about this framework is the minimal resource requirements:

  • ~400 bytes of memory for the static elements (linked list, SysTick, time implementation)
  • 4-5 instructions per job for initializing the NVIC
  • 2-3 instructions per queue element for initializing the asynchronous queue
  • 3-4 instructions + 4 bytes of stack space for a lock
  • 1-3 instructions for an unlock
  • 2-4 instructions for a pend/clear
  • 20-30 instructions per item in the queue for async

If you are working on a bare-metal ARM program with real-time concerns, crect is an RTOS alternative that provides protection against common multithreading concerns like priority inversion, deadlocks, and race conditions.

For more on crect:

EMB2

We discussed the EMB2 project in the November 2017 newsletter in the context of Industry Standard Multicore APIs. I am including EMB2 again here to emphasize its importance as a modern embedded systems framework.

EMB2 is a framework developed by Siemens and the University of Houston. EMB2 provides generic building blocks for building multi-core or multi-chip embedded applications, including basic parallel algorithms, concurrent data structures, and application skeletons. Since EMB2 is targeted for embedded applications, it provides soft-real-time support, predictable memory consumption (no dynamic memory allocations after startup), non-blocking APIs, and support for task priorities and affinities.

The framework utilizes the MTAPI abstraction layer, enabling EMB2 programs to be easily ported to new operating systems and processor architectures. Heterogeneous and distributed embedded programming is simplified by utilizing MTAPI. Developers can easily distribute work across processor cores, hardware accelerators, GPUs, DSPs, FPGAs, or networked devices.

The EMB2 base library is implemented as a C API with C++ wrappers. While the parallel algorithms, dataflow patterns, and concurrent containers are implemented in C++. C99 and C++03 are used as the implementation standard to provide maximum usability in the embedded world. C11 and C++11 are also supported.

If you are building a product which uses a multi-core processor, multiple processors, or hardware accelerators, EMB2 provides a solid and portable foundation that will enable your team to take full advantage of your system's hardware resources.

For more on EMB2:

Around the Web

The Industrial Internet Consortium published their IoT Security Maturity Model. If your team is working on an IoT device, this model will help you figure out where you need to be on the security spectrum and how to properly invest in the relevant security requirements.

Thinking about cutting corners or ignoring security in your IoT product? You might be featured in the IoT Hall of Shame.

Niklas Hauser, the author of modm, has two great articles for improving your embedded C++ skills:

Website Updates

The Archive now has a Guest Posts section so our readers can find our external publications.

Additional terms have been added to the Glossary: SRP, SFR, Cortex-A, Cortex-M, RTFM, NVIC

The Software References page was expanded with a variety of embedded and software development links.

Hardware rules of thumb were added to the Hardware References page.

The For Beginners page was expanded with C pointer references, code review articles, and introductory GitHub resources.

New Articles

These articles were published on our website in April:

I also shared some of my personal business experience with the Harpoon team in a Customer Spotlight.

These were our most popular articles in April:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. Jenkins: Configuring a Linux Slave Node
  4. C++ Casting, or: Oh No, They Broke Malloc!"
  5. std::string vs C-strings
  6. An Overview of C++ STL Containers
  7. Jenkins: Running Steps as sudo
  8. Demystifying ARM Floating Point Compiler Options
  9. Implementing Malloc: First-fit Free List
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!

-Phillip

April 2018: C & C++ Libraries We Like

Welcome to the April 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Seven exceptional C and C++ projects to consider for your next product
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Open Source Projects to Consider For Your Next Product

I’ve selected seven exceptional C & C++ projects that I want to share with you. Integrating high-quality libraries in our code base enables us to take new approaches, use safer techniques, or simply reduces the amount of code we need to write. I hate reinventing the wheel unless there’s a compelling reason, so I like to spend time researching libraries before diving into development. I hope you can find something to benefit your next product.

C++ Projects

I primarily focus on developing new embedded systems in C++. The language provides features that allow me to write safer code and to detect errors in compilation rather than at runtime. Of the seven project we’ll review today, 5 of them are C++:

  • NamedType
  • foonathan/type_safe
  • foonathan/memory
  • POCO C++ Libraries
  • Safer C++

NamedType

NamedType is a library written by Jonathan Boccara, a C++ developer and author of the FluentC++ blog. The NamedType library provides a simple interface for using strong types, which are not natively supported in C++.

We often utilize native types, such as int or double, to represent the values in our software. The general nature of these types means that we can easily make mistakes in our code. Consider a simple rectangle constructor:

Rectangle(double width, double height);

If you swapped the width or height in your code, the compiler would never be able to tell you.

double width = 10.5;
double height = 3.0;
Rectangle r(height, width); // wrong but compiles!

By using strong types in our interfaces, we can make our APIs more explicit and rely on the compiler to catch any mistakes:

// Create our strong types
using Width = NamedType<double, struct WidthTag>;
using Height = NamedType<double, struct HeightTag>;
//…
Rectangle(Width width, Height height); // new constructor
//…
Rectangle r(Height(3.0), Width(10.5)); // compiler error - type mismatch!
Rectangle r2(Width(10.5), Height(3.0)); // ok

The great news is that strong types are zero-cost when compiling with -O2 (or -O1 with GCC). It's time to make your APIs more explicit and catch errors during compilation.

For more on NamedType and strong types in C++:

type_safe

The type_safe library is developed by Jonathan Müller, a C++ library and author of foonathan::blog(). The type_safe library provides zero-overhead utilities to help catch bugs at compile time.

Features provided by the type_safe library include improved built-in types (ts::integer, ts::floating_point, and ts::boolean) which prevent dangerous implicit operations like signed-to-unsigned promotion. The type-safe library also provides "vocabulary" types to help us write more expressive code, such as object_ref (non-null ptr), array_ref (reference to an array), and function_ref (reference to a function). Other interesting concepts are also provided, such as deferred_construction, a wrapper which allows you to create an object without constructing.

Similar to NamedType, this library also supports strong types. Where NamedType provides a simple interface for creating strong types, the type_safe library requires explicit declaration of the attributes our strong type supports. The increased overhead for setting up new types is worth the safety provided by explicitly deciding what operations should be allowed. In the example below, our strong type only has addition and subtraction operations enabled:

struct meter
: strong_typedef<meter, int>, addition<meter>, subtraction<meter>
{
    using strong_typedef::strong_typedef;
};

Use the type_safe library to write expressive code and increase the number of errors caught during compilation.

For more on type_safe:

memory

The memory library is also developed by Jonathan Müller. This library provides an new STL-compatible C+ memory allocator called RawAllocator. The RawAllocator is similar to the standard Allocator but is easier to use. The library also provides a BlockAllocator type which can be used for allocating large blocks of memory.

The project includes a variety of implementations, adapters, wrappers, and storage classes, including:

  • new allocator
  • heap allocator
  • malloc allocator
  • memory pools
  • static allocator
  • virtual memory allocator
  • make_unique and make_shared replacements which allocate memory using a RawAllocator

We are excited about using this library in our next embedded project and gaining increased control over memory allocations.

For more on the memory library:

POCO C++ Libraries

The POrtable COmponents (POCO) C++ Libraries is a collection of C++ class libraries whose goal is to simplify the development of network-centric applications. POCO is similar in concept to the Java Class Library, .NET Framework, or Cocoa. The POCO libraries are highly portable, allowing you to easily compile and run your application on multiple platforms.

The full list of features provided by the POCO libraries are too numerous to list here. A small sampling of features:

  • Caching framework
  • Cryptographic & hashing libraries
  • Logging framework
  • HTTP server & client
  • SSL/TLS support through OpenSSL
  • POP3 & SMTP clients
  • SQL database access
  • Multithreading (basic threads, synchronization, thread pools, active objects, work queues)
  • Stream classes for Base64 and binary encoding/decoding, compression (zlib), line ending conversion
  • XML parsing & generation
  • Zip file manipulation

While the POCO libraries provide portability and a plethora of features, my favorite aspect is the emphasis on code quality, style, consistency, and readability.

I highly recommend reviewing the POCO libraries before beginning your next project - you can find a gem (or twenty) that will save you development time.

For more on POCO:

SaferC++

The SaferC++ library provides safer implementations for many native C++ types. The library provides features such as:

  • Data types that are designed for multi-threaded use and asynchronous access
  • Drop-in replacements for std::vector, std::array, std::string, std::string_view that provide improved memory safety
  • Drop-in replacements for int, size_t, and bool that protect against use of uninitialized values and sign-unsigned comparison issues (similar to type_safe)
  • Improved pointer & reference types with different compatibility and performance tradeoffs

SaferC++ is usable with embedded systems as long as your platform has a functional STL implementation. Exception behavior can be controlled for your platform by modifying the MSE_CUSTOM_THROW_DEFINITION macro.

Using the library does incur a performance penalty. However, SaferC++ elements can be disabled during compile time (i.e. replaced with the standard type equivalents). This allows users to enable debug and test builds to use safer-but-slower features without adding overhead to release builds.

Since the SaferC++ types provide added safety and can be disabled when performance matters, I highly recommend using their drop-in types to catch and eliminate possible errors when using STL types. The easiest way to get started with SaferC++ is to utilize the mse::vector and mse::array types in place of std::vector and std::array. These types will help you catch potential memory issues lurking in your software. The README provides further tips for making your code safer.

For more on SaferC++:

C Projects

While we’ve been heavily focused on C++-based embedded systems development for the past few years, I did find two exciting C projects that I want to share with you:

  1. CException, a lightweight exception library
  2. Checked C, an extension to the C language which provides better protections against common memory errors

CException

CException is a project released by Throw The Switch. CException is designed to provide simple exception handling in C using the familiar try/catch/throw syntax. I've been recently thinking about the downsides of error logging, so I am excited to see a lightweight exception library for C.

The exception implementation is kept simple by only allowing you to throw a single error code. No support for throwing objects, structs, or strings is included. The library can be configured for either single-tasking or multi-tasking which makes this project a good fit for embedded systems using an RTOS. CException is implemented in ANSI C and is highly portable. As long as your system supports the standard library calls setjmp and longjmp, you can use CException in your project. If you're looking for an exception library to use on embedded systems, CException is for you.

For more on CException:

Checked C

Checked C is a research project from Microsoft which adds static and dynamic (runtime) checking for errors such as buffer overruns, out-of-bounds memory accesses, and incorrect type casts.

The project is implemented as an extension to the C language. New pointer and array types are provided with the goal of allowing programmers to better describe intended pointer use and the range of memory that is pointed to. Developers can select between types with and without bounds checking, as well as between types that can or cannot be used in pointer arithmetic.

Since Checked C is an extension to the C language, you will need a compiler that supports it. Microsoft provides a port of clang and LLVM that support the extension.

CheckedC can help you identify and eliminate common memory errors which plague us as C & C++ developers. Even better, existing C programs compiled with a Checked C compiler will continue to work. Raw pointers (e.g. int *) remain unchecked and pointer arithmetic is still allowed.

For more on Checked C:

Around the Web

In the September 2017 newsletter I shared a series of posts detailing the engineering behind the Voyager missions. If you liked those articles, Voyager Mission Telecommunication Firsts presents another take on the outstanding engineering achievements of the Voyager missions.

If you enjoyed last month's focus on development processes, I recommend reading Making Valgrind Easy. By integrating Valgrind into your static analysis process, you can find and fix memory issues that your compiler or static analyzer won't catch.

I came across this article written by Michael Barr in 2009, titled Firmware Architecture in Five Easy Steps. Read this article before starting your next embedded project.

Jonathan Müller of foonathan posted Guidelines for Rvalue References in APIs. This article is recommended for advanced C++ developers and library authors.

Website Updates

Our "About" information has been condensed into a single page. We've doubled down on keeping the primary website focused on embedded systems content. We've created a new website dedicated to our consulting business.

A classic (and free!) introductory embedded systems book, Programming Embedded Systems, was added to the Beginners page. While the examples are slightly dated, the concepts are valid and can be applied to modern embedded systems.

The Glossary saw new additions, including LVDS, PLM, and MIPI standards.

The old "Open Source Software" page has been merged into the Libraries page.

New Articles

The Dark Side of Error Logging was published as a guest post on Arne Mertz's Simplify C++ blog.

These posts were published on our website in March:

  1. Improving Our Software With 5 Lightweight Processes You Can Adopt This Month
  2. Getting Started with the Snapdragon Flight: Driver Development
  3. Seeing Intermittent GitHub Clone Failures on Jenkins? Check Your Repo Size
  4. Safely Storing Secrets in git

These were the most popular articles in March:

  1. Circular Buffers in C/C++
  2. C++ Casting, or: Oh No, They Broke Malloc!"
  3. Installing LLVM/Clang on OSX
  4. std::string vs C-strings
  5. Jenkins: Configuring a Linux Slave Node
  6. An Overview of C++ STL Containers
  7. Implementing Malloc: First-fit Free List
  8. Demystifying ARM Floating Point Compiler Options
  9. Jenkins: Running Steps as sudo
  10. Creating and Enforcing a Code Formatting Standard with clang-format

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!

-Phillip

March 2018: Lightweight Processes to Improve Quality

Welcome to the March 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • 5 methods for improving software quality that you can implement this month
  • The keys to success when adopting new processes
  • Interesting links from around the web
  • Embedded Artistry website updates and popular posts

Improving Our Software With 5 Lightweight Processes You Can Adopt This Month

It's that time of year when the Barr Group releases their yearly Embedded Systems Safety & Security Survey results. Last year's results were eye opening, as nearly 50% of respondents reported not using static analysis and 36% reported that they do not perform code reviews. The 2018 results were no better:

  • 38% of safety-critical products don't comply with a formal safety standard
  • 43% of teams working on safety critical products don't perform code reviews
  • 41% of of teams working on safety critical devices don't perform regression testing (54% for IoT product teams)
  • 33% of teams working on safety-critical products don't perform static analysis (49% for IoT product teams)

These numbers are even more alarming given the fact that 25% of the reported "internet-connected devices" could kill or injure people if hacked. 22% of respondents mentioned that security for connected devices wasn't even on their to-do list. We're becoming increasingly connected, but our standards for safety, testing, and verification are not keeping pace. I want to be clear: this is not acceptable.

Many teams skip crucial development processes and justify them with scheduling pressure or by blaming the boss. There will always be scheduling pressure, so we must adjust our approaches or we will continue to flounder. Bugs are expensive in time, money, and morale. Debugging accounts for 50% of most project schedules. We all make mistakes. Anything we can do to keep bugs out of our code or catch them as early as possible will save money and time.

I've selected five simple processes to improve quality that your team can adopt over the next month. These processes are cheap or free, apply across languages and platforms, and best of all - they work. While each process requires a bit of time to get up and running, there is little-to-no maintenance involved in continuing to use them.

Here are five lightweight processes for improving code quality and identifying problems early:

  1. Fix all of your warnings
  2. Set up a static analysis tool for your project
  3. Measure and tackle complexity in your software
  4. Create automated code formatting rules
  5. Have your code reviewed

Fix All of Your Warnings

The first thing I do when working on a new project is fix all of the compiler warnings. It's amazing to me how developers will ignore warnings or rationalize their presence. Occasionally you even run into a team who will fight tooth and nail to prevent you from fixing them!

The compiler knows the programming language better than you ever will. You should not ignore the compiler when it is alerting you to an issue. The way you are using the language is dangerous and likely has unintentional side effects. Depending on the warning, you might be introducing undefined behavior into your software. That's not our idea of quality software.

If you have warnings in your code base, fixing them is one of the fastest ways to improve quality. You will fix bugs and flaws in your program, regardless of whether or not they are currently problematic.

For more on compiler warnings:

Set Up Static Analysis Support

Static analysis tools provide us with even better feedback than the compiler. Your compiler will happily allow some problematic cases which are legal in language, such as out-of-bounds pointer accesses or missing initialization values. Your analyzer will catch these problems, and also report red flags such as unused or redundant code. When used throughout the development cycle, your static analysis tool can help you catch & prevent latent problems even earlier than your testing cycle.

Some governmental and industrial organizations are starting to require static analysis data for certification processes. Companies such as PRQA provide tools that can check for compliance with safety critical standards in a variety of industries.

There are many free static analysis tools available and their commercial counterparts are also inexpensive (most are less than $1000). At Embedded Artistry, we use Clang's static analyzer alongside clang-tidy.

Here are some resources you can use to find a static analysis tool that fits your needs:

Measure and Tackle Complexity

By the time you've eliminated warnings on your project and cleaned up glaring problems exposed by your static analysis tool, you've already made significant progress with software quality. The next goal is to measure complexity in your software. Because highly complex functions tend to be hard to understand, test, and maintain, these functions are prime candidates for refactoring and simplification.

By using a metric to measure complexity, we have a quantitative way to evaluate our code and identify pieces that need special attention. We can see how our changes are impacting the code base over time and trigger automatic alerts and reviews whenever a threshold is exceeded. We can focus our code reviews on functions with high complexity scores, making sure they get the most of our limited attention. Metrics aren't perfect, but they increase our insight into our software quality.

These are the simplest and most popular metrics for measuring code complexity:

  • Lines of code (LOC): a count of the non-blank, non-comment source lines in a function, module, or project
  • McCabe cyclomatic complexity (MCC): provides a complexity score based on the number of branches (e.g. conditional statements)
  • Strict cyclomatic complexity (SCC, CC2): expands MCC by considering the number of conditions within each branch, which provides an approximation for the number of test cases needed for full coverage

These free tools will calculate complexity metrics for C/C++. We currently use Lizard at Embedded Artistry.

For more on software complexity:

Create Auto-formatting Rules

Automated code formatting might seem like a strange recommendation to put into the top five, but it serves three purposes:

  1. Automated formatting reduces a programmer's cognitive load by eliminating an entire category of details and decisions they need to keep in mind
  2. Automated formatting improves the quality of our peer code reviews (the next recommendation) by eliminating arguments about style
  3. Automated formatting is the first step toward implementing and enforcing a coding standard

Every team that I've encountered with a written style guide inevitably ignores those guidelines, and multiple programming styles run rampant. Instead of relying on developers to constantly keep an arbitrary set of rules in mind, we can automate the process to make it simple and impersonal. At the very least, it's worth eliminating the pointless, time-wasting arguments that cause friction within our teams.

We use clang-format on Embedded Artistry projects. Uncrustify and Astyle are other popular code formatting tools.

For more on automated code formatting:

Have Your Code Reviewed

Writers accept the fact that first drafts are generally garbage and need heavy editing. Before I send this newsletter out into the world, it has usually gone through 2-3 self-editing sessions and 1-2 peer reviews. Along the way, the newsletter is trimmed and restructured. The result is a much better product than the initial draft.

Yet, for some reason, programmers seem to think that perfect code is produced on the first try. The 2018 Barr Group survey results showed that 54% of IoT product teams don't perform regular code reviews. The survey results also show that a painfully scary 43% of teams working on safety-critical software don't perform regular reviews.

Perfect code on the first try might be possible if you're a prodigious programmer. But remember: even Hemingway had an editor. A second set of eyes can identify flaws that you missed in your first pass. Another developer may have different experiences that provide insight into the merits or risks of your approach. The architect on your team probably has input on how a module should interface with other pieces of the program. The "ego effect" also comes into play: knowing that our code will be presented and reviewed by another human can dramatically improve the overall quality. We will spend time cleaning up and checking the logic before putting code up for judgment.

Code reviews can waste time and become unproductive if poorly implemented. Best Practices for Peer Code Review provides some excellent tips for getting started. Notably, a lightweight review process is more efficient and practical than long, in-depth reviews with multiple developers. Even performing reviews on only 20-33% of the submissions provides benefits due to the "ego effect". While 20% may seem low, remember that we are aiming for achievable: reviewing 20% of the source code is definitely better than none.

I highly recommend implementing peer code reviews after setting up automated code-formatting. This helps constrain code review discussions by preventing them from devolving into style nit-picking. If you've set up static analysis, make sure the tools are used prior to code reviews.

For more information on code reviews:

The Keys to Success When Adopting a New Processes

When adopting new processes, it's important to focus on one at a time. Adopting new processes in stages ensures that you have time to correctly implement each new technique before moving on to the next one. By implementing too many changes at once, you are likely to overwhelm your team and evoke a mutiny.

If you're leading a team, it helps to find someone who is excited and can help you champion the idea. Empower that person so they can demonstrate the benefits of the new process to your team. Back them up when there is pushback. Change is always hard. Expect the resistance, but don't let it stop you.

The key to making new processes stick is to make them as automated as possible. There is never a case where it isn't worth the time it takes to automate a development process. Automation ensures that the process is easy to follow and always happens, rather than trusting individual contributors to remember to follow a process. Automation also makes the process less personal. The rules are clearly defined and are being enforced by a tool. Depersonalization helps us view the situation dispassionately, rather than as an attack on our abilities.

To recap, when you implement new processes:

  1. Adopt one new process at a time
  2. Empower a process champion on your team
  3. Automate!

Around the Web

BMW has issued a recall for 11,700 engines because they flashed the wrong firmware onto them.

In 2007, doctors replacing Dick Cheney's heart defibrillator ordered the manufacturer to disable its wireless capabilities out of concern for a hacker being able to trigger a fatal heart shock. We need to take the security of our connected devices, especially medical ones, seriously.

IT Hare published an extremely interesting analysis of operation costs in CPU clock cycles. Their data provides a great source of ballpark numbers for the relative computational costs of popular operations.

Michael Barr shared the 2018 survey results for programming languages used to develop embedded systems. C remains the main player with C++ in a distant second.

Website Updates

We've added two great introductory embedded systems resources by Embedded.fm to the For Beginners page. If you're just starting out with embedded systems, check out Embedded Wednesdays and Embedded Software Engineering 101.

The Glossary has been expanded with additional embedded systems terms.

The Software References page has been updated with a TLA+ reference for those interested in verifying that their algorithms work correctly.

Articles Published in February

These posts were added to the website in February:

  1. Implementing std::mutex with ThreadX
  2. Implementing std::mutex with FreeRTOS
  3. Refactoring the ThreadX Dispatch Queue to Use std::mutex
  4. Code Cleanup: Splitting Up git Commits in the Middle of a Branch
  5. Generating GStreamer Pipeline Graphs

These were the most popular articles in February:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. C++ Casting, or: Oh No, They Broke Malloc!"
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Implementing std::mutex With ThreadX
  7. Implementing Malloc: First-fit Free List
  8. Creating and Enforcing a Code Formatting Standard with clang-format
  9. Implementing an Asynchronous Dispatch Queue
  10. A Simple Consulting Services Agreement

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you wait on the next edition, check out the website or follow us on Twitter

Happy hacking!

-Phillip

February 2018: Spectre and Meltdown

Welcome to the February 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build better embedded systems. This newsletter is intended to supplement the website and covers topics not mentioned there.

This month we'll cover:

  • Two vulnerabilities that have rocked the computing world: Spectre and Meltdown
    • An overview of speculative execution
    • The Spectre vulnerability
    • The Meltdown vulnerability
    • What you can do today
    • Reliable sources for more information
  • Interesting links from around the web
  • Articles Published in January
  • Most Popular articles in January

A Tale of Two Vulnerabilities: Spectre and Meltdown

The new year did not get off to a great start for the electronics industry. A design flaw in modern processor architectures was exposed with the announcement of two critical vulnerabilities: Spectre and Meltdown. Both attacks are based on speculative execution, a technique in modern process design used to increase performance by pre-loading memory and future CPU instructions. Processor design has been largely focused on improving performance, and unfortunately the security implications of these improvements were not questioned. Spectre and Meltdown show that attackers can exploit speculative execution to access arbitrary memory locations.

Collectively, Spectre and Meltdown affect most processors that are on the market today. While Meltdown is a potentially patchable issue affecting most Intel and some ARM processors, Spectre can only be fully resolved by re-thinking our processor designs.

An Overview of Speculative Execution

Modern processors utilize instruction pipelines, effectively breaking up incoming instructions into sequential stages to keep the processor continually busy. Each pipeline stage handles its part of the current instruction, passes the result to the next stage, then handles the next instruction. This means that a processor effectively is working on multiple instructions in parallel.

The problem with instruction pipelines arises when a conditional branching decision is encountered. Consider the following simple statement:

if(x < 10)
    A();
else
    B();

The condition if(x < 10) must be evaluated before the processor knows whether it needs to call function A() or function B(). The pipeline stalls until the correct path is chosen, and valuable computational cycles are wasted.

Speculative execution is an optimization technique that tries to prevent those wasted computational cycles. Instead of stalling, the processor executes beyond the branch point (e.g. by calling A()) before it knows whether the branch will be taken. If the speculation was correct, then we gained the advantage of not wasting any computational cycles. If it was incorrect, the CPU discards the resulting state and continues executing on the correct path.

Note that the "speculated state" is not discarded until the correct execution path is known. This opens up the mis-speculation window, the time in which the CPU has speculatively executed the wrong code while not detecting that a mis-speculation has occurred.

Compounding with the mis-speculation window is caching, another performance optimization. When data is loaded into memory, caches are updated. If we need to refer to that memory again in the future, we can reduce the access time by fetching that information from the cache instead.

However, once the memory state is rolled back after a mis-speculation is detected, the data for the speculated state is still present in the cache. Clever attackers can take advantage of these speculative execution breadcrumbs to execute code and access memory that is not normally accessible.

There are currently three known variants of speculative execution vulnerabilities:

  1. Bounds check bypass (Spectre)
  2. Branch target injection (Spectre)
  3. Rogue data cache load (Meltdown)

At a high level, the Spectre exploits can be used to trick processors into running instructions they should not have run, granting access to arbitrary memory from another program's memory space. Meltdown applies primarily to Intel processors and can be used to access protected kernel memory in user space.

The Spectre Vulnerability

Spectre's name reflects the vulnerability's root cause (speculative execution) as well as the fact that it will haunt us for years to come. If your processor utilizes speculative execution techniques, you are probably affected. This includes the Intel, AMD, and ARM processors that power our personal computers, smart phones, and cloud servers. There are two primary variants of the Spectre vulnerability: branch target injection and bounds check bypass.

The branch target injection method relies on influencing how a processor's branch predictors operate. By influencing the branch predictors an attacker can control the speculative execution path and ensure that the malicious code is speculatively executed.

The bounds check bypass method takes advantage of speculative execution that occurs while the processor is checking if the targeted memory location is in-bounds. The processor will speculatively access out-of-bounds memory before the bounds check resolves. The attacker can read normally inaccessible memory by using a bounds check bypass combined with an intermediary program or module with better memory access privileges.

The end result is the same: processors inadvertently grant access to arbitrary memory from another program's memory space by running speculative instructions. After the code has been speculatively executed, an attacker can use the caches to reconstruct the target data.

The Meltdown Vulnerability

Meltdown is named for the fact that it "melts" normally enforced security boundaries between user memory and kernel (system) memory. Meltdown affects Intel, Qualcomm, and some ARM cores. Since Intel chips form the basis of many server platforms, most of the cloud service providers are affected by Meltdown.

Meltdown is only possible on processors which allow speculative execution across "privilege boundaries", such as the boundary which separates kernel memory from normal user programs. Normally, direct kernel memory access from user space is expected to fail with a page fault access error. However, certain processors might speculatively access the protected memory and use it for subsequent instructions prior to finishing the permissions check. If the permissions were not correct, a flag would be set and an exception thrown.

However, the kernel memory was still speculatively accessed and available in the cache during the mis-speculation window, allowing kernel memory to be read from user space.

What You Can Do Today

Your system is at risk if someone can run malicious code on your machine. This includes your personal computers, your phone, systems with multiple accounts, servers, cloud platforms, and virtualized environments. If your system is isolated from a network and only has a single user account, you probably don't need to worry about this exploit. If you're unsure of whether or not you need to take action, WindRiver's CEO provided a detailed framework for deciding how to respond.

If you are using an Intel chip or a cloud platform, make sure to stay up to date on patches for your system. The KPTI/KAISER patches for Meltdown have been applied to the latest Linux kernels but may need to be migrated to your specific kernel version. Similar patches have also been made for Windows and OS X. ARM has announced that some of its chips are affected by Meltdown, so check and see if your platform is affected.

Spectre is a trickier beast, especially since most modern processors are affected. A complete resolution is only possible through re-architecting the way processors are designed. We will be living with this threat for many years. Spectre-hardening software efforts are in progress, including ARM speculation barrier, MSVC mitigations, and LLVM mitigations.

Solutions for both of these vulnerabilities will be evolving over time, so be sure to stay up-to-date if you are affected.

Reliable References

The announcement has been surrounded with quite a bit of drama: CERT's initial recommended solution was to "replace CPU hardware", Linus Torvalds called Intel's patches "insane" and "complete garbage", and Intel admitted that its patches for Meltdown and Spectre are flawed.

Given all this drama, here are reliable sources you can refer to for more information:


Around the Web

For some non-Spectre-and-Meltdown news, check out the following articles.

Nordic Semiconductor has announced a new nRF91 chipset targeted for low-power IoT systems. The nRF91 System-inPackage (SiP) integrates an LTE cellular modem and transceiver, ARM Cortex-M33 processor, flash memory, RAM, and power management into a single package.

The Vancouver startup Riot Micro announced the RM1000, a low-power cellular baseband chip targeted for IoT applications.

Michael Barr proposes dropping the term "bug" in "Is it a Bug or an Error?". Jack Ganssle shared similar thoughts in "I've Never Had a Bug in My Code"

A new C++ programming language standard is now available.

Arne Mertz of Simplify C++ published a series on the topic of code reviews. The humble code review is one of the most powerful tools that teams can utilize to improve software quality and reduce errors. How can your team improve its review process?

  1. Code Reviews - Why?
  2. Code Reviews - What?
  3. Code Reviews - Preparation
  4. Code Reviews - How?
  5. Code Reviews - The Human Aspect

Articles Published in January

These posts were added to the website in January:

  1. Implementing an Asynchronous Dispatch Queue With FreeRTOS
  2. Getting Started with Snapdragon Flight: Dev Environment Setup & Useful Resources
  3. Implementing an Asynchronous Dispatch Queue with ThreadX
  4. Implementing malloc with FreeRTOS
  5. Jenkins: Configuring a Linux Slave Node
  6. Installing ROS on an NVIDIA Tegra TX2

Popular Articles

These were the most popular articles in January:

  1. Circular Buffers in C/C++
  2. Installing LLVM/Clang on OSX
  3. C++ Casting, or: Oh No, They Broke Malloc!"
  4. std::string vs C-strings
  5. An Overview of C++ STL Containers
  6. Implementing Malloc: First-fit Free List
  7. Creating and Enforcing a Code Formatting Standard with clang-format
  8. A Simple Consulting Services Agreement
  9. A GitHub Pull Request Template for Your Projects
  10. Implementing an Asynchronous Dispatch Queue

Happy hacking!

-Phillip