Welcome to the November 2018 edition of the Embedded Artistry Newsletter! This is a monthly newsletter of curated and original content to help you build superior embedded systems. This newsletter supplements the website and covers topics not mentioned there.
This month we'll cover:
- The Big Hack story and its ensuing drama
- A request for modern Embedded Linux resources
- Embedded articles from around the web
- Website updates
The Big Hack
At the beginning of this month, Bloomberg dropped a bomb by publishing "The Big Hack: How China Used a Tiny Chip to Infiltrate America's Top Companies". They followed the initial story with details about the software side of the attack. The Bloomberg article is a thrilling and dramatic read which has stirred up quite a bit of controversy.
If you haven't read the original story, here's a brief summary:
Between 2014-2015, Supermicro server motherboards had a small IC inserted onto the PCB, supposedly connected to the baseboard management controller (BMC). The chip allowed attackers to alter “the server’s Core OS so it could accept modifications and contact attacker-controlled computers for further instructions/code”. Supermicro subcontractors installed these chips at the behest of the People’s Liberation Army (PLA) of China. The software side of the picture involves two vectors: shipping Supermicro servers with outdated firmware that contained known security vulnerabilities, and releasing firmware updates with new vulnerabilities which would be installed after the boards were already in customers’ hands.
Unsurprisingly, the story has been vehemently denied by all parties involved:
- Apple published "What Businessweek Got Wrong About Apple", the VP of Information Security sent a letter to Congress, and Tim Cook called for Bloomberg to retract the story.
- Amazon also denied the story, but has since gone silent on the matter.
- Supermicro published a direct denial of the story. They were questioned by the US Senate and claimed that there was no evidence of the hack.
- The Chinese ministry of foreign affairs did not make an outright denial, but claimed that they are also a victim of supply chain attacks and "hope parties make less gratuitous accusations and suspicions".
- The Department of Homeland Security said that "at this time we have no reason to doubt the statements from the companies named in the story".
- The FBI director, as is their policy, did not comment on the possibility of an investigation other than saying "Be careful what you read".
Bloomberg was transparent about the companies' denials and published them on their website. Bloomberg then doubled down on the Big Hack by publishing a follow-up article stating that new evidence of hacked Supermicro hardware was found in a (unnamed) US telecom. Unlike the "tiny chip" on the motherboard, the reported hack involved an implant built into the Ethernet connector.
The Big Hack and the ensuing wave of denials have read like a fictional spy thriller. Following the commentary surrounding the story has been fun and enlightening:
- Joe FitzPatrick at Securing Hardware, one of the sources named in the story, shared his thoughts around hardware implants, how he'd execute such an attack, and how compromising the BMC firmware would be a much simpler approach than implementing the hardware hack.
- Bloomberg published the story a month after the NASDAQ delisted Supermicro.
- Supermicro shares plunged ~50% following the first announcement.
- 20+ people made suspicious short positions and sale orders just before the story broke, leading to calls for an SEC investigations and suspicion that this is a Supermicro hit piece.
- Matt Levine speculated on whether securities fraud would apply if companies falsely denied the story.
- Risky Business published a podcast digging into the story and interviewed Joe FitzPatrick, who said he feels “uneasy” about the story as published.
- Interestingly, Vice states that Joe FitzPatrick "told Motherboard in an online chat he was 'not surprised to see an example finally.'" This is an odd statement for someone claiming the story made them feel uneasy.
- Serve the Home analyzed the story and the BMC theory.
- Vice dug into supply chain vulnerabilities and included this interesting quote:
- “There are two possible stories here,” Matthew Green, associate professor at Johns Hopkins University, tweeted about the attack. “One is that there was an attack. The other is that a large swath of the National Security establishment is promoting the idea that there was an attack. Pick your poison.”
- Sup China takes a similar tack as Matthew Green and postulates that government sources may be trying to generate or amply mistrust of China.
- One Bloomberg opinion piece relates the hack to the growing conflicts between the US and China. Another opinion piece postulates on how hopelessly vulnerable global supply chains are - further stirring the pot in regards to China's business practices.
Now that a month has passed, we seem to be at an impasse. The lack of evidence of hacked hardware, a list of affected SKUs, or confirmation by other agencies casts doubt on Bloomberg’s story. The companies named continue to deny the story, and Bloomberg will not make a retraction, name sources, or provide further details. Other news agencies have been unable to independently confirm the story.
Regardless of whether the story is true or false, it highlights the vulnerability of our modern technological supply chains. I published my musings on modern supply chains and the feasibility of hardware tampering on the website.
A Request for Modern Embedded Linux Resources
I've received emails asking for recommendations for up-to-date Embedded Linux resources. We primarily focus on writing software for microcontrollers, so I have little experience with Embedded Linux. Most of the reference material that I've seen is at least a decade old or focuses on Raspberry Pi development using python libraries.
If you're an Embedded Linux developer who writes drivers or other low-level code, I'd love to hear about your favorite books, websites, and courses. Where do you turn to when you have a problem? What training do you take to improve your skills? What tools do you use most often?
You can respond directly to this email or send us a message on Twitter.
Around the Web
Another day, another security flaw announcement, this time for Amazon FreeRTOS.
Phil Koopman highlighted the cost of producing highly safety-critical software.
In the December 2017 edition of the newsletter we covered advances in chip design techniques. Read this IEEE article for another new chip design technique: Through-Silicon Transistors Could Make Stacking Chips Smarter.
Meeting Embedded shared a short interview with Dan Saks, a champion of embedded C++ development.
Mohammad Afaneh at Novel Bits continued publishing his Bluetooth Mesh tutorial series:
- Part 1: BT Mesh Concepts
- Part 2: Models, Scenes, Nodes, and BT Mesh Architecture
- Part 3: Provisioning and Security
- Part 4: nRF5 SDK for BT Mesh
- Part 5: Setting Up the Light Switch Mesh Demo Application
- Part 6: The Demo Application
- Part 7: Controlling the Network
Jack Ganssle finished publishing "Top 10 Reasons Embedded Projects Get Into Trouble". He finished his serious with two of the most common problems I see: "Quality Gets Lip Service" and "Unrealistic Schedules".
- 10: Not Enough Resources Allocated to a Project
- 9: Jumping into Coding Too Quickly
- 8: The Undisciplined Use of C and C++
- 7: Bad Science
- 6: Crummy analog/digital interfacing
- 5: Weak managers or team leads
- 4: Writing optimistic code
- 3: Poor resource planning
- 2: Quality Gets Lip Service
- 1: Unrealistic Schedules
The Glossary received another extensive update focused on manufacturing and supply chain terminology.
We updated our Technology Radar to include the Espressif ESP32 chipset. Saket Vora introduced us to this chip, which features 512kB SRAM, qSPI supporting 16Mbit external flash, low deep sleep current, and a superb datasheet. We're evaluating it for use with our embedded framework.
We published the following articles in October:
We also updated the following articles with new content:
These were our most popular articles in October:
- Circular Buffers in C/C++
- Jenkins: Configuring a Linux Slave Node
- Installing LLVM/Clang on OSX
- C++ Casting, or: "Oh No, They Broke Malloc!"
- An Overview of C++ STL Containers
- Jenkins: Running Steps as sudo
- Jenkins: Kick off a CI Build with GitHub Push Notifications
- Migrating from C to C++: NULL vs nullptr
- Implementing Malloc: First-fit Free List
Thanks for Reading!
Have any feedback, questions, suggestions, interesting articles, or resources to recommend to other developers? Simply reply to this email!