Mars Spirit Rover Filesystem Problems

30 March 2026 by Phillip JohnstonEighteen sols into its mission on Mars, the Spirit rover went dark: no modulated telemetry, missed communication windows, and the operations team spent several increasingly panicked days working with scraps of information (carrier signals, beep timing, uplink rates) to diagnose the problem from interplanetary distance. A design flaw in the DOS file system library caused RAM to fill up. When files were deleted from FLASH, the on-disk space was freed, but the RAM-resident data structures tracking those files were never released — they just marked deleted entries with 0xE5 and kept them allocated. So the …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Debugging Case Studies

9 January 2026 by Phillip JohnstonThis page collects documented debugging stories that we found interesting or useful. Table of Contents: Articles Audio Video Related Articles A Disappearing Service Processor by Laura Abbott of Oxide Computer Company During some of our first attempts at putting our next generation Cosmo sled into an Oxide rack, we would see the Service Processor drop off the network. This is a tricky situation to debug, as without network access we have limited insight into the state of the SP itself. A Gap in the TrustZone Preset Settings for the LPC55S69 / Oxide by Laura Abbott …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

AVM vs Steck: LGPL License Lawsuit Case Study

29 January 2025 by Phillip JohnstonGerman company AVM, which produces networking equipment, was sued by German software developer Sebastian Steck in 2023 for failing to provide complete source code under the lesser GPL (LGPL). AVM pulled a common move: they made source code available but left out key pieces. Steck’s initial complaint was that scripts and details for cross-compilation and installation onto the device were missing. Steck and AVM had several exchanges over a two year period, with Steck requesting more details and AVM disagreeing with Steck’s claim about obligations under the LGPL or that he should be able to …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Broken Stack Smashing Protection in GCC ARM Compilers

27 October 2023 by Phillip JohnstonThere have been several recent instances where Arm GCC toolchain compiler bugs rendered stack smashing protection incomplete or easily defeated. 32-bit Arm Error: Comparison Against the Address, not the Value We have written about implementing stack smashing protection for microcontrollers. Someone commented on that article pointing out that stack smashing protection in the GNU ARM compiler has been broken for quite a while: safe version seem to be up to 8.3.1, and after 10.2.1. Details about the problem can be found in the write-up linked in the comment: Faulty Stack Smashing Protection on ARM Systems. …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

RepoJacking Vulnerability

31 August 2023 by Phillip Johnston • Last updated 29 February 2024RepoJacking is a type of Supply Chain Attack that GitHub repositories can become vulnerable to. RepoJacking can occur when a GitHub user or organization changes its name. GitHub automatically creates links between older names and newer names, such that any uses of the older names will redirect to the new one. This is done to prevent dependencies from breaking when a rename occurs. However, the previous user name or organization name can now be used by others. If a new user or organization is created with the old name, …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Security Case Studies

28 August 2023 by Phillip Johnston • Last updated 24 February 2025In our careers, we will build relatively few systems. Our experience will be limited. If we want to expand our horizons beyond the narrow subset of systems we have worked on, we need to study the works of others. We collect case studies of systems and events that we believe are worth studying due to the lessons and insights they provide. These case studies specifically focus on security. Vulnerabilities can give you insight into how your systems can be abused to achieve some other end. Often, in a way …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

SolarWinds Supply Chain Attack

SolaWinds Corporation sells network management software to thousands of companies and government agencies. In 2019-2020, SolarWinds was subject to one of the most famous cases of a Supply Chain Attack through its product “Orion.” This attack is dubbed SUNBURST.

Researchers had been warning SolarWinds for years that their infrastructure was insecure, culminating in a November 2019 warning to SolarWinds that their FTP server was insecure, allowing hackers to upload malicious files to be distributed to SolarWinds customers. SolarWinds also had their Microsoft Office 365 account compromised, giving attackers access to emails.

The warned event came to pass. In December 2020, Actors working on behalf of the Kremlin pushed malware to roughly 18,000 organizations around the world using SolarWinds software. Follow-up payloads were sent to 10 U.S. federal agencies and 100 private organizations.

Attack Summary

Multiple vectors were used in this attack.

  • Microsoft-related Exploits
    • At least one reseller of Microsoft cloud services was compromised by the attackers, constituting a supply chain attack that allowed the attackers to access Microsoft cloud services used by the reseller’s customers.
    • Microsoft protocol NetLogon contained a “Zerologon” vulnerability, giving attackers access to all valid usernames and passwords in breached Microsoft networks.
    • These credentials were used to access the Microsoft Office 365 email accounts.
  • SolarWinds Exploits
    • Attackers accessed the SolarWinds build system. They modified software updates to plant remote access tool malware into SolarWinds updates. When users updated, the malware was installed, which would stay dormant for two weeks before communicating with the attackers’ command-and-control infrastructure to indicate success and provide access to the backdoor. Communications were designed to mimic legitimate SolarWinds traffic.
    • Because the SolarWinds product was connected to customers’ Office 365 accounts, attackers were able to access emails at customer sites.
    • Email accounts were leveraged to gain access to SAML tokens, allowing attackers to masquerade as legitimate users.
    • Attackers exfiltrated data of interest from customer sites.
  • Easily guessed default passwords, including “password” and “solarwinds123”

Multiple U.S. government agencies were confirmed to be breached through this attack, including:

  • Department of State
  • Department of Homeland Security
  • Department of the Treasury
  • Department of Energy
  • National Telecommunications and Information Administration
  • National Institute of Health
  • National Nuclear Security Administration

2023 SEC Lawsuit

On 30 October 2023, the U.S. Securities and Exchange commission announced a lawsuit against “software company SolarWinds Corporation and its chief information security officer [CISO], Timothy G. Brown, for fraud and internal control failures relating to allegedly known cybersecurity risks and vulnerabilities.” The SEC is seeking disgorgement of “ill-gotten gains,” civil monetary penalties, and permanent barring Brown from acting as an officer or director for any company that issues securities.

The SEC complaint accuses SolarWinds of securities fraud due to overstating its cybersecurity practices and understating known risks. In essence, SolarWinds touted their strong security practices

Quote

The Security Statement was materially misleading because it touted the Company’s supposedly strong cybersecurity practices. For example, that statement asserted that SolarWinds created its software products in a “secure development lifecycle [that] follows standard security practices including vulnerability testing, regression testing, penetration testing, and product security assessments.” And the Security Statement claimed that SolarWinds’ “password policy covers all applicable information systems, applications, and databases [and we] enforce the use of complex passwords.” It also stated that SolarWinds had “[a]ccess controls to sensitive data in our databases, systems, and environments [that are] set on a need-to know / least privilege necessary basis.” All those statements were materially false and misleading.

The SEC complaint points out that SolarWinds knew about their cybersecurity risks for years, and several employees at the company (including CISO Brown) repeatedly complained in writing about the shortcomings. However, the SEC points out, “rather than address these vulnerabilities, SolarWinds and Brown engaged in a campaign to paint a false picture of the company’s cyber controls environment.”

Cybersecurity-wise, the SEC’s primary complaints include SolarWinds’:

  • failure to consistently maintain a secure development lifecycle for software it developed and provided to thousands of customers
  • failure to enforce the use of strong passwords on all systems
  • failure to remedy access control problems that persisted for years

The following quotes and complaints from the lawsuit filing show that the situation was not good at all.

  • In an April 2017 email to the newly hired CIO, a SolarWinds employee expressed surprise that things “like ‘default passwords’ are [still] plaguing us when the product has been in the market [this long,]” explaining, “[m]any of these vulnerabilities seem pretty well amateur hour.” As an example, the employee noted one product for which the default password was “password.” Senior InfoSec Manager E testified that having a default password of “password” is a “poor security practice.”
  • A January 2018 email to senior managers bluntly admitted that the Security Statement’s Secure Development Lifecycle (“SDL”) section was false, and described a “simple” scheme by which, rather than amend the Security Statement to make it accurate, SolarWinds would conceal the present falsity of the representations and work toward making them true eventually: “I’ve gotten feedback that we don’t do some of the things that are indicated in the [Security Statement’s SDL section]. I want to make sure that you all have an answer to this. The simple response is: There is improvement needed to be able to meet the security expectations of a Secure Development Lifecycle. We will be working with teams throughout 2018 to begin incorporating the SDL into their development lifecycle.”
    • A later email notes that as of June 2020, Orion was still not following an SDL process.
  • An April 2018 audit shared with SolarWinds’ CIO identified multiple critical systems that did not comply with the password policy. The audit found systems where “shared SQL legacy account login credentials [were] used,” contrary to the Security Statement’s claim that SolarWinds “require[s] that authorized users be provisioned with unique account IDs.”
  • In June 2018, SolarWinds Network Engineer D2 identified a “security gap” relating to SolarWinds’ remote access virtual private network, which allowed access from devices not managed by SolarWinds. Network Engineer D warned that this setup was “not very secure” and later explained that someone exploiting the vulnerability “can basically do whatever without us detecting it until it’s too late” which could lead to a “major reputation and financial loss” for SolarWinds.
  • Illustratively, in October 2018, the same month that SolarWinds conducted its Initial Public Offering through a registration statement with only generic and hypothetical cybersecurity risk disclosures, Brown wrote in an internal presentation that SolarWinds’ “current state of security leaves us in a very vulnerable state for our critical assets.”
  • An August 2019 presentation warned that “[a]ccess and privilege to critical systems / data is inappropriate.”
  • Presentations in March and October 2020 highlighted “[s]ignificant deficiencies” in SolarWinds’ access controls.
  • In a July 2020 presentation, Brown warned about threat actors’ familiarity with a critical SolarWinds software platform, noting that the threat actors “[k]now how to deploy software, shut off backup, etc.”
  • In a July 2020 email to Brown, a member of the Engineering team described being “spooked” by activity at a SolarWinds’ customer. Brown agreed that the incident was “very concerning” and continued, “As you guys know our backends are not that resilient and we should definitely make them better.”
  • A September 2020 Risk Acceptance Form flagged for Brown and others “the risk of legacy issues in the Orion Platform” and warned “[t]he volume of security issues being identified over the last month have outstripped the capacity of Engineering teams to resolve.
  • In instant messages sent in November 2020, SolarWinds’ Senior InfoSec Manager E expressed his own disgust with the Company’s security posture, lamenting, “[W]e’re so far from being a security minded company. [E]very time I hear about our head geeks talking about security I want to throw up.
  • In November 2020, a SolarWinds Information Security employee sent an instant message to Senior InfoSec Manager E with a link to a list of vulnerabilities in the Orion platform stating, “The products are riddled and obviously have been for many years.” That same month, a SolarWinds’ network engineer complained, “We filed more vulnerabilities then [sic] we fixed. And by fixed, it often means just a temporary fix…but the problem is still there and it’s huge. I have no idea what we can do about it. Even if we started to hire like crazy, which we will most likely not, it will still take years. Can’t really figure out how to unf**k this situation. Not good.

The SolarWinds misrepresentations of their security practices and risks caused an overvaluation of the stock. Once details of the attack were revealed, there were significant losses.

Quote

On December 14, 2020, the day it filed the Form 8-K first announcing the SUNBURST attack against the Orion platform, SolarWinds’ stock price dropped more than 16%. It dropped at least an additional 8% the next day. The stock price continued to drop and lost approximately 35% of its value by the end of the month as SolarWinds disclosed more details of the SUNBURST attack, and as news outlets reported that internal sources had warned SolarWinds for several years about the Company’s cybersecurity risks and vulnerabilities.

References

For More on the 2019-2020 SolarWinds attack:

For more on the 2023 SEC Lawsuit:

Case Study: MSI UEFI Signing Key Leak

25 May 2023 by Phillip Johnston In our recent panel dicussions, such as Over-the-Air Updates and Fleet Management at Scale, we touched on the topics of code signing, security keys, and key management. The whole strategy of code signing hinges upon protecting your keys. If they are compromised, anyone will be able to create and sign an image that can be successfully installed on your systems. You can mitigate this risk by creating a key rotation and/or revocation strategy, allowing you to stop compromised keys from being usable. Failing to incorporate these capabilities puts you at significant risk. In May …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

The Update Nightmare: Bricking Devices in the Field

25 May 2023 by Phillip Johnston • Last updated 6 November 2025 In several of our past panel discussions, such as Over-the-Air Updates and Fleet Management at Scale and From Concept to Launch: What It Takes to Build and Ship a New Device, we’ve touched on the scary possibility of bricking devices in the field through an update. The possibility of catastrophic failure drives our release processes, quality enforcement mechanisms, OTA validation efforts, and fleet management capabilities. The costs of a failed update are significant. Depending on the size of your business and scale of the problem, this situation may …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Nexx Product Line Vulnerabilities

25 April 2023 by Phillip Johnston This story serves as a prime example of the reason we support security-related regulatory efforts. There are a number of fundamental practices you should focus on with respect to security and privacy. Regulatory efforts and their supporting standards often reference this set of fundamentals. Examples include: Validate inputs Authenticate access Don’t use universal default passwords or other hard-coded credentials Encrypt sensitive data And yet, Nexx’s line of garage door controllers, alarms, and plugs have failed on all of those counts. Nexx’s devices can be used to control garage doors, home security alarms, and smart …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.