July 2019: Security, Security, Security!

Welcome to the July 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Phillip's interview on Embedded.fm
  • Atomic clocks in space
  • Stuxnet: ten years later
  • Embedded systems security resources
  • Embedded news from around the web
  • Updates to the Embedded Artistry Website

Embedded.fm Interview

Last month, Embedded.fm interviewed Phillip. Elecia, Chris, and Phillip discussed consulting, writing, learning, code reviews, rules of thumb, Boeing 737 MAX, and implementing C++ threading types with an RTOS.

Check out Embedded.fm episode 290!

Atomic Clocks in Space

We love learning about technological developments with space exploration, since they involve wonderful examples of highly reliable embedded systems.

In June, NASA’s Jet Propulsion Laboratory (JPL) announced that they were creating a deep-space atomic clock. This clock is a toaster-sized device that aims to facilitate navigational efforts for future space missions. The first clock was launched into Earth’s orbit at the end of June for a one-year trial.

You can learn more about the Deep Space Atomic Clock with these JPL links:

A Look at Stuxnet, Ten Years Later

June marked the 10th anniversary of the first Stuxnet infection, which targeted Siemens PLCs and was responsible for damaging Iran's nuclear program. Stuxnet caused centrifuges at the nuclear plant to spin out of control while operator screens reported nominal values, leading to systems failures, asset damage, safety concerns, and a national security fiasco.

It's interesting to look back on Stuxnet with the context of today's continual onslaught of security vulnerability announcements and IoT botnet attacks. What have we learned in the ten years since Stuxnet demonstrated the capability of vulnerabilities to wreak havoc on physical systems? The overall state of device security hasn't noticeably improved, and reports of hacked devices reaching into the physical world abound.

Security can no longer be an afterthought. Even if your product isn’t connected to a nuclear reactor, vulnerabilities are exploited with disastrous results. For more on Stuxnet:

For more on Stuxnet:

Embedded Systems Security Resources

A reader reached out to us on LinkedIn and requested recommendations for books and resources discussing embedded systems security. As we just mentioned, security is an area that we all need to focus on when we design and build our embedded systems. If you have recommended books or links that discuss embedded systems security, let us know! We will be collecting recommendations and publishing a post dedicated to security resources. Here’s what we’ve collected thus far.

Phil Koopman has a collection of lectures discussing embedded systems security:

The following books relate to embedded systems security:

Here are articles with general introductions to embedded systems security concepts:

These articles focus on common pitfalls with embedded device security:

Around the Web

We've grouped our July reading recommendations by these categories:

  • General
  • Software
  • Security
  • Hardware

General

Elecia at Embedded.fm wrote a great post about Giving Feedback prompted by our conversation on the podcast.

Bunnie comments on the trade war with China and its potential impact on Open Source Could Be a Casualty of the Trade War.

All About Circuits published a helpful introduction to the Advanced Microcontroller Bus Architecture (AMBA), which is the de facto standard for SoC design.

Software

If you're interested in the intersection of machine learning and embedded systems, a reader on LinkedIn shared this link for running TensorFlow Lite on a microcontroller.

The Memfault team published an excellent article on digging into firmware code size. If you're trying to reduce your binary's size, this article is a must read.

The Memfault team also published an excellent article on writing linker scripts for firmware.

Colin Walls wrote about mixing C and C++ in an embedded application. If you're interested in trying out C++ on an existing embedded product, try one of Colin's two approaches.

Burkhard Stubert added another article to his Speaking CAN series, this time analyzing write buffer overflows.

Watchdog timers are a staple element of embedded systems. Alexandru Lazar shared some Watchdog Timer Anti-patterns that we should keep in mind when building our systems.

Thread Group announced version 1.2 of the Thread wireless protocol. Stacey Higginbotham provided a summary of the changes

For a great example of applying std::variant to create a state machine, check out this C++ article: Space Game: A std::variant-Based State Machine by Example.

Security

Appropriately for the 10-year anniversary of Stuxnet, we saw an inordinate amount of articles related to security.

EE Times highlighted that the IoT security gap is widening: we're adding more devices and sending more data, but IoT devices are still a prime hacking target.

Designing Hardware for Data Privacy highlights the need to develop a thread model for our devices and to implement the appropriate countermeasures.

Enhancing Privacy and Security in the Smart Meter Lifecycle provides a look at a variety of techniques that can be used to improve IoT device security.

How Many Layers of Security Do You Have? argues that designers should consider two guiding principles: defense in depth and principle of least privilege.

Will the Real Root of Trust Stand Up? discusses the concept of a "root of trust".

Hardware

Strange Parts recorded a fantastic factory tour of the PCBWay factory in Shenzen: Inside a PCB Soldering Factory - in China. If you are curious about how circuit boards are manufactured, check out this video.

Dangerous Prototypes shared a helpful app note discussing magnetometer placement in mobile devices.

Dangerous Prototypes also shared a helpful app note discussing a method for using two accelerometers to make rotational measurements, which is useful if you already have two accelerometers and don't want to update your product to add a gyroscope.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our newsletter.

Website Updates

Stickers are now available in the Embedded Artistry store. You can help support the website by purchasing a sticker.

We've renamed the "Embedded C++" and "Migrating from C to C++" categories to work around a Squarespace link generation issue.

We added additional acronyms to the Glossary.

We added new book recommendations and links to For Beginners, Hardware References and Software References.

We updated the following articles:

New Articles

We published the following articles in June:

These were our most popular articles in June:

  1. Creating a Circular Buffer in C/C++
  2. For Beginners
  3. std::string vs C-strings
  4. Demystifying Microcontroller GPIO Settings
  5. Installing LLVM/Clang on OSX
  6. C++ Casting, or: "Oh No, They Broke Malloc!"
  7. Improving Your Callback Game
  8. Jenkins: Configuring a Linux Slave Node
  9. An Overview of C++ STL Containers
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip & Rozi

June 2019: Programmer vs Engineer, Coroutines, Report CI

Welcome to the June 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Programming vs engineering
  • Coroutines
  • Report CI
  • Embedded news from around the web
  • Updates to the Embedded Artistry Website

We also want to share some exciting news with you! One of our articles was published in the June 2019 issue of the Software Quality Professional Journal. You can read The Boeing 737 MAX Saga: Lessons for Software Organizations for free if you register for an account with ASQ. Or you can email us and we will send you a copy.

Programmer vs Engineer

We recently came across an article from The Atlantic titled "Programmers: Stop Calling Yourselves Engineers".

The article begins by highlighting the use of "engineer" for programming roles:

In the Silicon Valley technology scene, it’s common to use the bare term “engineer” to describe technical workers. Somehow, everybody who isn’t in sales, marketing, or design became an engineer. “We’re hiring engineers,” read startup websites, which could mean anything from Javascript programmers to roboticists.

What's wrong with programmers co-opting the term "engineer"? The article points out that:

Traditional engineers are regulated, certified, and subject to apprenticeship and continuing education. Engineering claims an explicit responsibility to public safety and reliability, even if it doesn’t always deliver.

The fact that programmers don't have a similar set of responsibilities is alarming because:

Computing has become infrastructure, but it doesn’t work like infrastructure.

And aside from computing infrastructure that we've come to rely on, we are building more and more software-intensive complex systems:

It’s no accident that the most truly engineered of software-engineering projects extend well beyond the computer. Autonomous-vehicle design offers the most obvious contemporary example. When Google designs self-driving cars, it musters its own computational systems, like mapping and navigation. But it also integrates those into a world much larger than browsers and smartphones and data centers. Autonomous vehicles share the roads with human-driven cars, pedestrians, and bicyclists. Those roads are managed, maintained, and regulated. Self-driving cars also interface with federal motor-vehicle standards and regulations, along with all the other material demands and foibles of a machine made of metal and plastic and rubber rather than bits. Engineering addresses complex, large-scale systems.

We're sympathetic to the article. The fundamental complaint seems to be that companies are co-opting the term engineer for glory, but they are merely profiteering raiders who are avoiding the rigor required of engineering professionals. We can't deny this assessment given many of the requests we receive from new hardware companies looking to strike it rich as quickly as possible.

Fortunately, there seems to be a slowly growing wave of increasing professionalism that may stand to counter the reckless creation of software systems. Jack Ganssle tells a chilling tale about the evolution of fire safety standards and equates the situation to software. He, and many others, also call for eliminating the term "bug" and replacing it with "error".

Jack’s fundamental question still rings true, especially in light of the Boeing MCAS debacle and loss of 346 lives. How many more software-related catastrophes will we experience before we demand a fire code for people working on critical software systems and infrastructure? If programmers want to be considered engineers they ought to be held to the same standards as engineers in other specialities.

Coroutines

We've recently been digging into coroutines for our C++ framework, which is well-timed because they are slated to land in C++ with the upcoming C++20 release. Although they are just being added to C++, the coroutine concept exists in a variety of languages and can be applied even without language standard support (see libaco for a C implementation, and Klemens Morgenstern's talk for a lightweight implementation for C++ on ARM controllers.

As a brief overview, a coroutine is a function which:

  • May or may not have its own stack (depending on the implementation)
  • Can interrupt its own execution at specific suspension/yield points
  • Can be resumed later

Coroutines provide an asynchronous processing alternative to threads. Instead of having a scheduler which interrupts threads, coroutines interrupt themselves. Asynchronous processing is achieved by manually sequencing slices of coroutine functionality.

As always, there are tradeoffs: coroutines give us deterministic asynchronous processing, but they do not provide true concurrency that can take advantage of multiple processors or cores. This set of tradeoffs is generally attractive for embedded systems, as embedded systems designers are often more concerned with determinism than taking advantage of concurrent processing.

The links we list below have helped us immensely in our study of coroutines. We'll be applying them to our embedded C++ framework to simplify some complex areas of asynchronous processing that currently uses hard-to-follow chains of callbacks (these notes from IT Hare demonstrate the complexity problems we are facing). Avoiding chains of callbacks will greatly simplify our asynchronous code.

For more on coroutines in general:

For more on C++ coroutines:

Report CI

Klemens Morgenstern has released a new GitHub integration called Report.ci. Report.ci allows you to report automated unit test results and compiler logs to GitHub from your chosen continuous integration (CI) system. Rather than having to dig around in the build server logs to figure out why a build or test failed, the results can be uploaded to Report.ci and shown directly in the GitHub pull request (under both "Checks" and "Files Changed").

Here are some examples of the service in action:

Report.ci currently supports 28 unit test frameworks and compiler log annotations for many popular languages and compilers. Other capabilities of the service include Badges, Annotation merging for cross-compilation scenarios, and mapping annotations so they appear at the code under test (rather than the test code itself).

Report.ci is free for public GitHub projects to use. Give this plugin a try if you want to see test results and compiler logs directly within your GitHub pull requests.

Around the Web

We've grouped our May reading recommendations by these categories:

  • Software
  • Hardware
  • History

Software

Niall Cooling continued his series on the difference between semaphores and mutexes with Part 2: The Mutex & Mutual Exclusion Problems.

François Baldassari, CEO of Memfault, shared his take on the program startup process in From zero to main(): Bare metal C.

Mohammad Afaneh of NovelBits published a guest post on the Interrupt blog on Debugging Firmware with GDB. If you are unfamiliar with using GDB,

Fotis Chatzinikolaou shared helpful communication protocol design tips in Designing Communication Protocols, Practical Aspects.

Hardware

Maya Posch published an article on Hackaday which describes the physics behind soldering and IC packaging.

Arch Martin at AVX wrote an excellent app note describing the basics of decoupling.

We found an app note from Renesas on eye safety protection guidelines for products using infrared LEDs.

History

Take a look at the die of Intel's 8087 floating point coprocessor chip

All About Circuits published an article describing the early history of the oscilloscope.

Hiring Embedded Engineers?

Is your company hiring embedded systems hardware/software engineers? Send us a short job ad with a link to the full job description. We will be happy to include it in our newsletter.

Website Updates

We added new links to the Software References, Hardware References, and For Beginners pages. The new links include a list of all Embedded.fm episodes, links to video playlists, and a review of free Python courses.

A link to our curated YouTube video playlists has been added to the "Resources" menu.

We corrected typos in our article: Ditch Those Built-In Arrays for C++ Containers.

New Articles

We published the following articles in May:

These were our most popular articles in May:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. Installing LLVM/Clang on OSX
  4. Demystifying Microcontroller GPIO Settings
  5. Migrating from C to C++: NULL vs nullptr
  6. C++ Casting, or: "Oh No, They Broke Malloc!"
  7. An Overview of C++ STL Containers
  8. Jenkins: Configuring a Linux Slave Node
  9. Jenkins: Running Steps as sudo
  10. Improving Your Callback Game

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

May 2019: GPS Rollover Day, Historical Source, and YouTube Playlists

Welcome to the May 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • GPS rollover day problems
  • Historical source code
  • Embedded Artistry YouTube playlists
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

GPS Rollover Problems

6 April 2019 was "GPS Rollover day". The current GPS standard uses a 10-bit counter to keep track of the number of weeks elapsed since 6 January 1980. Since it's a 10-bit counter, the max count is 210-1, or 1023, weeks. On the 1024th week, it will rollover to week 0.

If you’re keeping score, you’ll be surprised that this was a big deal at all: we already experienced a GPS rollover on 21 August 1999. Many GPS equipment manufacturers understand the rollover requirements and designed their systems to accommodate it. However, we knew that some GPS receivers would be unable to make the transition gracefully. These receivers would reset their date to the last epoch (21 August 1999). Once the epoch is reset, these devices were expected to give the wrong time and position. In the worst case devices might lock up permanently.

Companies had plenty of time to prepare for this event and firmware updates were sent out for many affected devices. But as expected, GPS Rollover Day still impacted many systems:

Let this be a warning to system designers: your systems will remain in use longer than you expect. Spend time investigating and handling edge cases, especially foreseeable ones like a recurring 1024-week GPS rollover.

Historical Source Code

Contemporary programmers are lucky: we live in a world where historical and influential program source code is available for us to review. However, most programmers only learn and study the programs they have worked on themselves. We rarely take the time to study historical works, and programming courses don’t typically spend any time on the subject.

We believe that software developers should review influential source code. This is similar to architects studying influential building designs (and critiques of those designs). Rather than repeating the same mistakes over-and-over, we should study the great works that preceded us and learn from and build upon their lessons.

If you're interested in studying historical source code, we've collected links in a new article on our blog: Programmers: Let's Study Source Code Classics.

If you have recommendations for classic source code or commentary, simply respond to this email and we will add them to the list.

Embedded Artistry YouTube Playlists

Over the past three years, we've shared dozens of YouTube videos on Twitter and in our newsletter. Most of those videos were shared only once. If you weren't looking at just the right moment, you missed the link. We've decided to collect our video recommendations in a YouTube channel.

We've created the following playlists and populated them with videos we've recommended in the past:

There are already many quality embedded systems playlists on YouTube. You can find our playlist recommendations in the Saved Playlists view and on the channel home page.

We will feature other channels that regularly publish quality embedded systems content, such as:

We will be continually expanding these playlists as we discover new videos. If you have videos to recommend to the embedded community, simply respond to this email.

Around the Web

We've grouped our May reading recommendations by these categories:

  • Industry News
  • Product Development
  • Firmware
  • Hardware

Industry News

Microsoft bought ThreadX, which is our favorite RTOS. Now we see that major tech companies are individually sponsoring an IoT/RTOS solution: Microsoft has ThreadX, Amazon has FreeRTOS, Intel has Zephyr, and Google has Fuchsia.

The Zephyr Project made their first long-term support (LTS) release. Here is more information about the plans for Zephyr LTS:

Product Development

Not all signal processing applications need a fully-fledge Digital Signal Processor (DSP) All About Circuits provides helpful tips for selecting a microcontroller to use instead of a DSP chip for signal processing applications:

These relate to another helpful All About Circuits article published last month: How to Choose the Right Microcontroller for Your Application.

Firmware

All About Circuits published a high-level Introduction to Real-Time Embedded Systems

Niall Cooling cleared up common misconceptions about mutexes and semaphores in Mutex vs Semaphore: Pt 1.

David Bergfeld walks through a thorough state machine design exercise in Software Design of State Machines.

Avinash Aravindan concluded his Flash 101 series with Error Management in NAND Flash. If you haven't followed along, here are the first five articles:

  1. NAND Flash vs NOR Flash
  2. The NOR Flash electrical interface
  3. The NAND Flash electrical interface
  4. Types of NAND Flash
  5. Errors in NAND Flash

Colin Walls Some More Embedded Software Programming Tips

Hardware

Microcontroller datasheets are massive tomes which can intimidate new engineers. All About Circuits shared two helpful guides for navigating processor datasheets:

Lee Teschler at Microcontroller Tips discussed the Hidden Pitfalls of IoT Antenna Design

All About Circuits provided a helpful guide to Understanding Amplitude Quantization Error for ADCs.

EETimes gave us a look at the power systems engineering for satellites in Taking Care of Power Systems for Space Missions.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all our open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We expanded the Glossary with additional terms:

  • Bottom-half handler
  • Top-half handler
  • LMA
  • VMA
  • XIP

We expanded the For Beginners page with Miro Samek's Modern Embedded Systems Programming Video Course and links to help readers learn RTOS concepts.

We updated the following articles with corrections and new content:

New Articles

We published the following articles in April:

These were our most popular articles in April:

  1. What can software organizations learn from the Boeing 737 MAX saga?
  2. Creating a Circular Buffer in C/C++
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. Migrating from C to C++: NULL vs nullptr
  6. Demystifying Microcontroller GPIO Settings
  7. Jenkins: Configuring a Linux Slave Node
  8. An Overview of C++ STL Containers
  9. C++ Casting, or: "Oh No, They Broke Malloc!"
  10. Jenkins: Running Steps as sudo

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

April 2019: Boeing 737 MAX and Mirai

Welcome to the April 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The grounding of the Boeing 737 MAX
  • The return of the Mirai botnet
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

Grounding the Boeing 737 MAX

One of the largest news stories over the past month was the grounding of Boeing 737 MAX-8 and MAX-9 aircraft after an Ethiopian Airlines crash resulted in the deaths of everyone on board. This is the second deadly crash of involving a Boeing 737 MAX. A Lion Air Boeing 737 MAX-8 crashed in October 2018, also killing everyone on board. As a result of these two crashes, Boeing 737 MAX airplanes are temporarily grounded in over 41 countries, including China, the US, and Canada. Boeing also paused delivery of these planes, although they are continuing to produce them.

We've been following the Boeing 737 MAX story closely. It serves as an interesting case study on software and systems engineering, human factors, corporate behavior, and customer service.

We started writing about the Boeing 737 MAX investigation as part of the newsletter and ended up with an 8,000 word essay on the saga. We cover the problem as it is currently understood, discuss contributing factors which are common across engineering organizations, and focus on lessons we learned from researching the crashes. I hope that all of you will take the time to read the article. There are important lessons we can all learn from this tragedy.

*Note: Both the Lion Air and Ethiopian Airlines crashes are still under investigation. Ultimately, everything you are reading about these crashes and that we discuss in the article is still in the realm of speculation. However, the situation is serious enough and well-enough understood that Boeing is addressing the problem immediately.*

Mirai Returns

In 2016, the Mirai botnet was discovered after it was used in some of the largest DDoS attacks. You were probably impacted by Mirai when the Dyn attack rendered many popular sites inaccessible, such as GitHub, Twitter, Reddit, Netflix, Paypal, Fox News, CNN, and Amazon.

Mirai is malware that targeted networked IoT devices running Linux. Many companies ship devices with default usernames and passwords enabled. Mirai takes advantage of this fact by continuously scanning for vulnerable devices and using an expansive list of factory default logins. If successful, the victim’s IP and login credentials was sent back to a collection server. Devices remain infected until they are rebooted, but they are quickly re-infected if the login is unchanged.

While the original author and his compatriots pled guilty to the Mirai-related crimes, the Mirai source code was released and used by other groups.

A new variant was discovered in January and announced this month. This new variant includes new credentials and exploits. The target list has been expanded, encompassing routers, network storage devices, IP cameras, and TVs. WePresent WiPG-1000 Wireless Presentation systems and LG Supersign TVs, both of which are marketed to businesses, are two devices which are specifically targeted. Enterprises are juicy targets for botnets because they typically have access to larger bandwidths, which the attackers can use to increase the impact of DDoS attacks.

As embedded systems creators, the only thing we can do to help stem the flood of compromised devices and botnet attacks is to set our own houses in order. If you are building an embedded system, change the default passwords and ensure that you are applying security patches for your devices. If a device on your business's network has a security flaw or default login that cannot be changed, you must remove it or firewall it.

For more on the new variant of Mirai:

For more on the original wave of Mirai attacks:

Around the Web

There was a lot of activity in March, so we've grouped our reading recommendations by category:

  • Product Development
  • Firmware
  • Hardware

Product Development

Kerry Scharfglass gave an absolutely wonderful talk on Design for Manufacturing. Kerry shares many of the lessons he learned the hard way about factories and manufacturing test software. If you're working on an embedded device and haven't yet been to a factory, this is a great introductory lesson and can save you a ton of pain.

The Verge published a look at the original iPhone development kit. Apple prototype development boards were the best that I've ever used as a firmware engineer - use some of these design ideas for your next dev board.

The terrible nature of IoT security has prompted Congress to introduce a bill titled The Internet of Thing Cybersecurity Improvement Act. Lawmakers are looking to legislate a bare minimum security standard that all IoT devices would have to meet. Stacey Higginbotham shared her take on the IoT bill.

Nike's self-lacing shoes stopped working after an update, even with the manual buttons. This reminds us to be extremely careful when designing products that require an Internet connection to function.

Firmware

Burkhard Stubert took a small survey of companies running demos of neural networks on microcontrollers.

Neil Tan went further and wrote a guide for running a simple neural network on MCUs. Neil's guide will enable you to train and load models on a microcontroller.

Jacob Beningo published "Minimizing energy consumption in Amazon Free RTOS applications" on Embedded.com.

Piotr Grygorczuk shared his implementation of C++ threading using FreeRTOS (std::thread, std::mutex, std::condition_variable, std::chrono, std::sleep_for, etc.).

Rud Merriam tweeted to us about his Embedding C++ series on Hackaday, where he explores the use of C++ on embedded systems.

Andreas at AndWass C++ recently started his blog. He's published three great embedded-related C++ articles:

  1. Multitasking 1 of n
  2. A Basic Scheduler
  3. Exploring A Driver Concept

Hardware

ST Micro published a great technical note addressing frequently asked questions related to Reset and Supervisor ICs.

Not sure what IoT standard to choose for your product's radio? This article from Microcontroller Tips attempts to predict the winners of the IoT radio standards war.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all our open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We fixed broken links on our Resources for Beginners page.

We updated our Technology Radar with the latest changes in our technology stack.

We updated the following articles with new content:

New Articles

We published the following article in March:

These were our most popular articles in March:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. C++ Casting, or: "Oh No, They Broke Malloc!"
  4. Installing LLVM/Clang on OSX
  5. Jenkins: Configuring a Linux Slave Node
  6. Jenkins: Running Steps as sudo
  7. Migrating from C to C++: NULL vs nullptr
  8. Jenkins: Kick off a CI Build with GitHub Push Notifications
  9. An Overview of C++ STL Containers
  10. Demystifying Microcontroller GPIO Settings

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

March 2019: Deprecating Volatile

Welcome to the March 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • How we get to main()
  • Deprecating volatile
  • Volatile load/store
  • Embedded news from around the web
  • Embedded job postings
  • Updates to the Embedded Artistry Website

What Happens Before Main?

We're working on a new article which describes the steps our programs take to get to the main() function. Whether we are running our programs on a PC or an embedded system, there are plenty of behind-the-scenes steps needed to get our system into a usable state.

At a high level, Embedded Artistry systems tend to match the following general pattern:

  1. Initialize the processor & external memory
  2. Setup the C/C++ runtime (reloc, BSS, constructors)
  3. OS initialization
  4. Hardware platform initialization (configure IO, start drivers)
  5. Jump to main()

We would love to hear from you: what steps does your embedded system take to get to main()?

Your feedback will help us refine our general boot model.

Deprecating Volatile

A recent C++ proposal for deprecating the volatile keyword has surfaced. This may surprise our readers, because as Michael Caisse said, "volatile is the embedded keyword."

The original intent of the volatile keyword in C89 is to suppress read/write optimizations:

No cacheing through this lvalue: each operation in the abstract semantics must be performed (that is, no cacheing assumptions may be made, since the location is not guaranteed to contain any previous value). In the absence of this qualifier, the contents of the designated location may be assumed to be unchanged except for possible aliasing.

The problem with its use in C++ is that the meaning is much less clear, as it is mentioned 322 times in the current draft of the C++ Standard.

One problematic and common assumption is that volatile is equivalent to atomic. All the volatile keyword denotes is that the variable may be modified externally, and thus reads/writes cannot be optimized. This means that the volatile keyword only has a meaningful impact on load and store operations.

Where programmers run into trouble is using volatile variables in a read-modify-write operation, such as with the increment (++) and decrement (--) operators. These operations create a potential for a non-obvious race condition, depending on how the operation is implemented in the compiler and platform. Chained assignments of volatile values (a = b = c) are another problematic volatile scenario. Is b re-read before storing the value to a, or not?

To address these usage problems, the authors created a separate proposal for volatile_load() and volatile_store() functions. We suggest an initial implementation of these functions in the next section.

Here's the list of proposed changes with the "deprecation" of volatile:

  1. Continue supporting the time-honored usage of volatile to load and store variables that are used for shared memory, signal handling, setjmp / longjmp, or other external modifications such as special hardware support.
  2. Deprecate (and eventually remove) volatile compound assignment op=, and pre / post increment / decrement -- ++.
  3. Deprecate (and eventually remove) volatile-qualification of member functions. Don’t change volatile-qualification of data members.
  4. Deprecate (and eventually remove) partial template specializations involving volatile, overloads on volatile, and qualified member functions for all but the atomic and numeric_limits parts of the Library.
  5. Deprecate (and eventually remove) volatile member functions of atomic in favor of new template partial specializations which will only declare load, store, and only exist when is_always_lock_free is true. Preserve most volatile free function overloads for atomic.
  6. Deprecate (and eventually remove) non-reference and non-pointer volatile parameters. Deprecate (and eventually remove) constas well as volatile return values. References and pointers to volatile data remain valid.

A rationale for each of these is provided in the original paper, "§3.6 Why the proposed changes?".

For more on the proposals related to deprecating the volatile keyword:

A Volatile Load/Store Implementation

Prior to the proposal, Embedded Artistry implemented volatile_load<T>() and volatile_store<T>() template functions to encourage better volatile behavior in our programs.

Since developers commonly misinterpret volatile access as atomic, seemingly innocuous statements can have surprising effects on program behavior if you are depending on atomic access.

volatile int i = 2; //probably atomic
i++; //not atomic ...

Using volatile_load() and volatile_store() ahead of standards changes will force a better usage pattern:

auto r = volatile_load(&i);
 r++;
 volatile_store(&i, r);

You can use these functions to refactor your programs and control volatile use cases. While this implementation does not meet the proposed specification, it's a step toward cleaning up the volatile keyword.

Here is our implementation of volatile_load() and volatile_store():

#include <cassert>
#include <type_traits>

/** Read from a volatile variable
 *
 * @tparam TType the type of the variable. This will be deduced by the compiler.
 * @note TType shall satisfy the requirements of TrivallyCopyable.
 * @param target The pointer to the volatile variable to read from.
 * @returns the value of the volatile variable.
 */
template<typename TType>
constexpr inline TType volatile_load(const TType* target)
{
    assert(target);
    static_assert(std::is_trivially_copyable<TType>::value,
        "Volatile load can only be used with trivially copiable types");
    return *static_cast<const volatile TType*>(target);
}

/** Write to a volatile variable
 *
 * Causes the value of `*target` to be overwritten with `value`.
 *
 * @tparam TType the type of the variable. This will be deduced by the compiler.
 * @note TType shall satisfy the requirements of TrivallyCopyable.
 * @param target The pointer to the volatile variable to update.
 * @param value The new value for the volatile variable.
 */
template<typename TType>
inline void volatile_store(TType* target, TType value)
{
    assert(target);
    static_assert(std::is_trivially_copyable<TType>::value,
        "Volatile store can only be used with trivially copiable types");
    *static_cast<volatile TType*>(target) = value;
}

Around the Web

Our February reading recommendations are grouped by category as follows

  • General Software
  • Firmware
  • Hardware
  • Product Development
  • C++

General Software

John Regehr collected samples of ASCII art that explains source code. If you're interested in adding ASCII art to your own code documentation, try out Buttersquid.ink. Buttersquid provides a link in the generated text so you can make updates to the diagram in the future.

We've talked about Spectre and Meltdown in past editions of our newsletter. IEEE published an informative article reviewing how the two attacks actually work.

The "Small Change" Fallacy makes us think we can skip full validation because a change is "small". All changes, no matter how innocuous, deserve to be treated seriously by running through the full code change process.

Jack Ganssle republished an old April Fool's Day joke: "Goto considered needed'.

Firmware

Jack Ganssle discussed the impending failure of older GPS devices due to an overflow problem.

Segger has launched the Smash-V2 compression algorithm, which is targeted for embedded systems. Compression requires 900B of ROM, and decompression requires 500-800B. No working RAM is required. Information is available on the Segger website.

Dan Luu analyzed the comparative performance of least-recently-used (LRU) cache eviction and random cache eviction. It turns out that random eviction performs pretty well under certain scenarios.

Arduino launched their own IoT cloud platform.

Mohammad Afaneh published two articles on prototyping with the nRF52840 USB dongle:

All About Circuits has published a set of articles introducing processor architectures:

Hardware

Japanese scientists have developed a MEMS energy harvester which can turn vibrations into power.

"Ensuring Fail-safe Data Storage in Battery Powered IoT Sensor Nodes" provides a strategy to address a common concern in battery-powered embedded systems.

The race is on to find an alternative to SRAM due to energy inefficiencies that are highlighted by the increase in data that the computing industry is managing. Part one of the series describes some of the problems with SRAM.

EE Times discussed the US-China trade crisis and the fallout for the chip industry.

Not sure what the difference is between slew rate and rise time? EDN clarifies the two concepts in "Slew Rate and Rise Time: Not Quite the Same".

Product Development

Pete Staples of Blue Clover Devices was on The Hardware Entrepreneur podcast. He discussed pitfalls with the NPI process and their new production line tool.

Bunnie Huang shared a talk on Supply Chain Security, a topic we've discussed in a previous edition.

C++

Jonathan Müller put together a great reference for when you need to declare different special member functions.

Rainer Grimm, a C++ trainer, addresses common template misconceptions and surprises.

Embedded Job Postings

iRobot is hiring for several embedded software roles in Boston, MA and Pasadena, CA. Please check out all of their open jobs here: https://irobot.wd5.myworkdayjobs.com/iRobot and email Chris Svec (csvec at irobot.com) if you'd like to learn more about building robots that help make people's lives easier.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We added additional links and book recommendations to both the Software References and Hardware References pages.

For Beginners was updated with additional links.

We updated the following articles with new content:

New Articles

We published the following article in February:

These were our most popular articles in February:

  1. Creating a Circular Buffer in C/C++
  2. std::string vs C-strings
  3. Jenkins: Configuring a Linux Slave Node
  4. C++ Casting, or: "Oh No, They Broke Malloc!"
  5. Installing LLVM/Clang on OSX
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. Jenkins: Running Steps as sudo
  8. Migrating from C to C++: NULL vs nullptr
  9. Demystifying Microcontroller GPIO Settings
  10. An Overview of C++ STL Containers

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

February 2019: Bluetooth 5.1 and Peer Code Review

Welcome to the February 2019 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • The new Bluetooth 5.1 specification
  • Phil Koopman's peer code review best practices
  • Embedded news from around the web (there was a lot of activity this month!)
  • Embedded job postings
  • Updates to the Embedded Artistry Website

Bluetooth 5.1 Released

The Bluetooth SIG released Bluetooth version 5.1 in January. The Bluetooth 5.1 release adds new features and capabilities to Bluetooth Low Energy (BLE).

Direction finding is the primary feature of the Bluetooth 5.1 release. The new direction finding capabilities include both Angle of Arrival (AoA) and Angle of Departure (AoD) measurements. Location services, such as asset tracking, item location, proximity marketing, and indoor way-finding, will use the new BLE direction finding capabilities. For products to take advantage of the new direction finding features, multiple antennas are required: 2+ on receivers for AoA method, and 2+ on transmitters for AoD method. BLE stacks will need controller-level updates to support the new features.

Other features of the 5.1 release include:

  • GATT Caching enhancements to reduce connection times in unbonded devices
  • Randomized advertising channel indexing to improve packet collision avoidance
  • Periodic advertising synchronization transfer
  • HCI support for Debug Keys in LE secure connections
  • Sleep clock accuracy update mechanism
  • ADI field in scan response data
  • Interaction between QoS and Flow Specification
  • Host Channel Classification for Secondary Advertising
  • Allow the SID to Appear in Scan Response Reports
  • Specify the behavior when rules are violated

For more on Bluetooth 5.1:

Phil Koopman on Peer Code Reviews

Peer code review is a code quality practice that we encourage all teams to adopt. This month, we listened to Phil Koopman's lecture series on Peer Code Review. This lecture series is full of actionable information which can help teams looking to adopt a formal peer code review practice. The series also serves as a source of new ideas for teams already performing reviews.

One area we want to highlight is Phil's rules for successful peer reviews:

  • Review everything that your team writes down:
    • Documentation
    • High-level design
    • Detailed design
    • Code
    • Test plans
  • Inspect the item, not the author
  • Don't get defensive - nobody writes perfect code
    • If you are getting defensive as the author, leave the room
  • Find problems; don't fix them in the meeting
    • Identify the symptom or technical deficiency of the problem as specifically as possible
  • The word "should" is not appropriate in a peer review
  • No debates on style
    • Enforce conformance to style guide
    • Do not debate whether the style guide is correct

Phil expands upon these rules with his peer code review best practices:

  • Formal reviews (Fagan inspections) optimize bugs found per dollar
    • Formal reviews find more bugs than informal reviews, and they find them more efficiently
    • Informal reviews typically find < 10-50% of defects
    • Online-review tools are OK, but they are not substitute for formal in-person review meetings
  • Target 10% of total project effort to find 50% of bugs
  • Also review documentation and designs - not just code
  • Review documentation and design first to find issues before coding begins
  • Use a checklist to find the most bugs
  • Target 1 peer review per day so you don't burn out reviewers
  • Schedule peer reviews regularly
    • 2-3 reviews per week as the project progresses
  • Limit meetings to 1-2 hours
    • After 2 hours, review effectiveness drops off
  • Target 150 lines of non-comment source code per hour for review
  • Keep a queue of items pending review; pull off queue for a scheduled review
  • Run static analysis tools before review meetings
    • Static analysis tools do not eliminate the need for peer code review

Phil states that formal peer code reviews should capture > 50% of defects. If your team is not finding sufficient bugs during code reviews, listen to the lecture series and try applying a formalized review process. All of us can improve the quality of our peer code review by applying these straightforward rules and best practices in our organizations.

Here are additional publications on peer code reviews by Phil Koopman:

Here are Embedded Artistry publications discussing peer code reviews:

Around the Web

There was a lot of activity in January, so we've grouped our reading recommendations by category:

  • Firmware
  • Hardware
  • C
  • C++

Firmware

Jack Ganssle shared insights into how we can improve our firmware development processes.

Researchers in Japan studied how tree frogs regulate their communications to avoid croaking over each other frog in the colony. Their findings have implications for a future network packet collision prevention scheme.

Low-powered IoT devices with limited memory capabilities will have to eventually support IPv6. 6LoWPAN will make that happen.

Japan plans to hack 200 milion IoT devices in February to compile a list of vulnerable devices. This effort further highlights the fact that IoT devices, not people, are now the security weak link.

The Multicore Association has released a new version of the Software-Hardware Interface for Multi-Many Core (SHIM). Version 2 of the specification introduces new features around power consumption properties, inter-connect contention descriptions, custom instructions, vendor extensions, and more.

Hardware

EDN gave us a look at NASA Apollo electronics from the 1960s.

Samsung shipped the first 1 terabyte NAND chip.

For years we have faced a constrained DRAM supply with rising prices. Now we are dealing with over-supply and a forecasted 20% price crash.

Nanopower announced their nP-BLE52 module, which is a System-in-Package (SiP) combining an nRF52832 with a custom power management IC. The custom power management IC enables the SiP to put the SoC into sleep mode, cut power, and wake it up at a pre-set time and restore the pre-sleep state. Sleep mode current consumption is spec'd at 10nA, making this processor ideal for low-power and long-life embedded applications.

Apple has filed a patent for a new version iBeacon, which will utilize ultra-wide band radio instead of BLE.

The electronics industry shipped over 1 trillion devices last year, including integrated circuits and optoelectronics, sensors, and discrete (O-S-D) devices.

All About Circuits has been publishing reviews of sensors, in case you need inspiration for your next project:

C

Feabhas published an article describing How to use C structs to access memory mapped peripheral registers. Many vendor SDKs and frameworks use this technique for interacting with processor peripherals.

C++

C++ can be a confusing language, especially when details such as lvalues and rvalues come up. This practical post on C++ moves is for those developers who don't know (or don't care) what rvalues are.

C++11 added smart pointers to our arsenal, but it's not always clear how we should move them into and out of functions.

As is commonly heard, const is a contract. It's important to understand exactly what const should communicate at different points in our programs.

Embedded Job Postings

Cisco Meraki is hiring for a Firmware Features Software Engineer in San Francisco. The features engineer will architect and implement features across the software stack, spanning from device drivers and routing code all the way up to Cisco's distributed, cloud-hosted backend.

Hiring Embedded Engineers?

Is your company hiring for embedded systems roles? Send us a short (< 100 words) job ad with a link to the description and we will be happy to include it in our newsletter.

Website Updates

We've added code syntax highlighting to our website. Please let us know if you notice any problems with code samples after the update.

We expanded the Glossary with additional terms.

We updated the following article with new content:

New Articles

We published the following article in January:

These were our most popular articles in January:

  1. Circular Buffers in C/C++
  2. std::string vs C-strings
  3. Jenkins: Configuring a Linux Slave Node
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. Jenkins: Running Steps as sudo
  8. Demystifying Microcontroller GPIO Settings
  9. An Overview of C++ STL Containers
  10. Migrating from C to C++: NULL vs nullptr

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip

January 2019

Happy New Year!

Welcome to the January 2019 edition of the Embedded Artistry Newsletter. This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.

This month we'll cover:

  • Highlights from 2018
  • The Embedded Template Library
  • A code "self-review" exercise
  • Embedded news from around the web

2018 in Review

We covered a lot of ground with our newsletter in 2018. We focused on four major topics:

  1. Improving Quality
    1. Lightweight Processes to Improve Quality
    2. Testing Embedded Systems
  2. Libraries suitable for embedded systems
    1. C and C++ Libraries We Like
    2. C++ Embedded Frameworks
    3. Three more useful embedded projects
  3. Changes to C++ that might impact embedded developers
    1. C++ standardization meeting in Rapperswil
    2. New Freestanding C++ proposal
    3. An update on Freestanding C++ & scoped enumerations proposal
  4. Security Flaws & Hacks
    1. Spectre & Meltdown
    2. Spectre Returns
    3. The Big Hack

2018 was a productive year for Embedded Artistry:

On a fun note: according to our Twitter analytics data, 99% of our followers love “dogs”, with “tech news” and “space & astronomy” following closely behind. Our team feels the same!

Embedded Template Library

We've been hard at work on our C++ embedded systems framework. A critical design requirement is that the framework core components do not utilize dynamic memory allocation. Unfortunately, this excludes most of the STL containers, as they get storage from the heap by default.

Rather than reinventing the wheel, we spent some time searching for existing solutions to this problem. We feel lucky to have discovered the Embedded Template Library (ETL), an open-source project created by John Wellbelove at Aster Consulting Ltd.

The ETL provides fixed-size alternatives to STL container types. The ETL containers specify a maximum capacity at compile-time and make no calls to malloc/free/new/delete. As an added bonus, the containers are all cache friendly: storage for all container types is allocated in contiguous blocks. The ETL also provides additional utilities and frameworks that are useful for embedded projects. The library can be safely used in conjunction with the STL.

The ETL will be included in our embedded framework and will be seen in our code samples for future articles. You can find additional ETL links and information on our website.

Code Self-Review Exercise

Code reviews are one of the essential practices for improving software quality. Aside from identifying issues with code before its deployed, reviews expose developers to new techniques, approaches, and patterns of thought.

As a consultant, it’s hard to enlist help for internal code reviews. As such, I’ve been testing other strategies to improve the code I write. For example, I use automated processes to check for problems or style inconsistencies, and I run through a checklist of common design mistakes before I merge in any new code changes.

Recently, I’ve been noting mistakes I’m prone to make and screening for them before submitting pull requests. I learned about this exercise in "Social Effects of Peer Review.". Here's a summary of the exercise from the article:

Try this experiment sometime: For one week, make a log of every error you make. Every misspelling in e-mail, every time you accidentally close the application you’re working in, every time you have a bug in your code, even every syntax error you make. Don’t get too specific; keep each item in the list high-level and use hash marks to count how many times you make that type of mistake.

For now, just imagine you’ve done this. As you might guess, certain patterns emerge. A few of the items have an awful lot of hash marks next to them. In fact, by the end of the second day you might have been getting tired of counting some of them!

And that annoyance might live in the back of your mind. You might even start thinking about it consciously. Pretty soon you’ll anticipate the mistake and prevent yourself from making it. Some of those will come easy, others will take work to eliminate. But eventually you develop habits that prevent that type of error completely. If you frequently forget to close parenthesis in your code, perhaps you’ll enable the feature in your code editor that closes parenthesis for you. If you close applications by accident, maybe you’ll pause just a few seconds every time you’re about to hit the close button. Maybe you’ll look up how to spell common words or learn how to input your common mistakes into the auto-correct facility in your e-mail editor. Over time you become more productive and more efficient. You’re working faster and smarter. Not through some self-help seven-step program, just by observing yourself a little more carefully and systematically than you’re used to.

Noting down all of the mistakes you make can also be a humbling experience. Give it a try this week as a New Year’s resolution and see how much you improve by just paying more attention to yourself.

Around the Web

MQTT and CoAP are ubiquitous in IoT applications. Unfortunately, Trend Micro reported that these messaging protocols are insecure. You can read their full report, "The Fragility of Industrial IoT's Data Backbone".

While we're on the topic of vulnerabilities, this article by The Register discusses the software bugs and software quality.

All About Circuits shared an embedded project that we love: How to Make Your Umbrella into a Weather Station with a Particle Photon.

All About Circuits also posted an introduction to MEMS systems. Since many components in our embedded systems utilize MEMS technology, we recommend familiarizing yourself with how it works.

This article provides a nice summary of how different industries are leveraging IoT devices.

IEEE Spectrum published an article describing sensors that are suitable for medical applications.

Orbit Fab launched an experiment to the ISS to test cube satellite refueling technology.

Ellisys has created a video series that teaches developers about Bluetooth. Videos include: Intro to Bluetooth Low Energy, Generic Access Profile (GAP), Advertisements, Connections, Generic Attribute Profile (GATT), Power Consumption, Security, Bluetooth 5 and IoT, and Intro to Bluetooth Mesh.

Jee Labs published a six-part series which introduces developers to the CAN bus:

  1. Intro
  2. Access
  3. STM32
  4. JeeH API
  5. Demo
  6. Single-wire

Website Updates

We updated the Embedded Artistry Technology Radar to reflect the changes in various technologies and techniques that are under assessment.

We also updated the following previously published articles with new and corrected content:

New Articles

We published the following articles in December:

These were our most popular articles in December:

  1. Circular Buffers in C/C++
  2. Jenkins: Configuring a Linux Slave Node
  3. std::string vs C-strings
  4. Installing LLVM/Clang on OSX
  5. C++ Casting, or: "Oh No, They Broke Malloc!"
  6. Jenkins: Kick off a CI Build with GitHub Push Notifications
  7. An Overview of C++ STL Containers
  8. Jenkins: Running Steps as sudo
  9. Migrating from C to C++: NULL vs nullptr
  10. Implementing Malloc: First-fit Free List

Thanks for Reading!

Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!

We’d also love to hear how the code review experiment goes for you.

While you're waiting for our next edition, check out the website or follow us on Twitter.

Happy hacking!

-Phillip