Mature Teams Prioritize End-to-End Interactions

27 March 2023 by Phillip JohnstonEmbedded systems are increasingly complex, especially those connected to the internet. In many cases, they are better viewed as one part of a larger distributed system involving the device, one or more phone applications, backend servers, fleet management, CI/CD servers, and more. Many teams do not approach product development with this distributed nature in mind. The “old ways” are followed instead: first, make all of the individual components work on their own, then integrate them together. The firmware team will work in their own world until the system is mostly completed, as will the app …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Processes that Aid Embedded Systems Development

14 July 2022 by Phillip Johnston • Last updated 4 February 2026Every embedded systems project and organization is different, but there are a consistent set of processes and practices that are valuable for the majority of embedded software projects. Mature embedded software organizations can be identified by their investments in these types of processes. Version PCBs Version Software Version Data Storage, Protocols, and Schemas Develop the riskiest or most critical part of the system first Prioritize End-to-End Interactions Exclusive Use of the Customer-Facing Firmware Update Mechanism Use a defined part numbering scheme for meaningful artifacts (including documents!) Use source control …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Reviews in Software Engineering

9 May 2022 by Phillip Johnston • Last updated 14 July 2022200-299 Embedded Artistry/250 Writing Projects/250.25 Software Engineering/250.25.21 Reviews in Software Engineering/250.25.21 Reviews in Software Engineering.md

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Versioning Software

20 January 2022 by Phillip Johnston Last updated 11 February 2026Versioning software projects is important, and especially so on embedded software projects: every device is likely to have a different software version flashed on it (especially during development), and there will be a range of software versions flashed onto customer devices at any given time. Because every binary is slightly different in some way or another, in order to debug the system we need to access the proper debug symbols, addresses, and change logs for that correspond to the version we are working with. For these reasons, we need the …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Versioning PCBs

We consider hardware versioning an essential device-side capability and process for embedded systems development. Every PCB used in an embedded system is bound to have multiple revisions throughout the product’s development cycle, and it is important that we can easily identify the revision we are currently working with – especially from within the software system.

The version number for a PCB should be incremented every time a new version is released, no matter how trivial the changes may seem.

Version Silkscreen on PCB

The PCB version should be printed on the board via silkscreen so that engineers and developers can unambiguously determine the version. Silkscreen is preferred to a sticker because stickers often come off, and a board may very well be mislabeled.

Version Listed on Schematic

At a minimum, print the PCB version on the schematic’s front page. Ideally, the version will be on every page of the schematic. This makes it quite easy for anyone on the team to check whether the schematic they are looking at refers to the PCB they have in hand.

Software-Interpretable Versioning

While the PCB silkscreen and schematic versions aid developers, they do not give the software access to a method to determine what hardware revision it is running on.

Some PCB revisions will be agnostic to software and merely involve tweaking some board design parameters. However, many revisions end up being significant enough to necessitate corresponding software changes: pinouts are changed, peripheral devices are changed, features are added/removed, etc.

In the “ideal” case, all of the will hardware can be immediately scrapped and the software will only need to support newer hardware. However, these ideal case rarely plays out. Because development hardware is often in limited supply due to its high production cost, teams often need to support multiple hardware revisions concurrently. Likewise, we may already be selling our devices, and now we need to maintain support for existing devices as well as the new variant. If we have a way to read the hardware version from software, we have the ability to support multiple variants with a single revision. We can also use the revision to access the proper firmware variant in the event that we maintain different binaries for different variants.

Frustratingly, the majority of embedded systems projects we work on still do not provide a method for the software to determine the hardware revision. This feature isn’t considered until it is too late, and there are multiple variants that need to be supported with no good way to distinguish them in software. At this point, the problem becomes one of build management: we must manually ensure that existing devices are flashed with the proper software version. This is an error-prone process

Use GPIO Encode PCB Version

The simplest and most common technique for encoding PCB revision so that software can access is by encoding a version using GPIO. The software will read the three GPIO lines and interpret them as a 3-bit binary version number. The GPIO will be configured in software with internal pull-ups. The version number in hardware is then controlled by placing resistors that connect the pin to GND.

Note

The same system works if the polarity is reversed as well. The directionality of the system and the value of the resistors are usually driven by power concerns (to minimize leakage current).

Version GPIO 2 GPIO 1 GPIO 0
1 0 0 1
2 0 1 0
3 0 1 1
4 1 0 0
5 1 0 1
6 1 1 0
7 1 1 1
8 0 0 0
Note

Some teams use 0b000 to represent version 1, and 0b111 will represent version 8, and some just make 0b000 version 0. In other cases, we have seen teams reverse the ordering such that they count down instead of up.

Three GPIO lines are often sufficient, as they provide a possible set of 8 versions before we need to reset. We have worked on teams that used only two GPIO lines due to GPIO constraints, but we have always found this insufficient: almost every system we have worked on has gone through four revisions before or after product launch. Four GPIO lines gives you much more wiggle room, but we have rarely been through that many revisions. In our opinion, it is better to re-start the version numbers once production hardware is ready, scrapping support for all development hardware at that time. Version increments after that point apply to production hardware changes.

Include High-Z as a Pin State

Most GPIO have a high-Z state, and you could leverage this to get 3 bits per pin instead of 2.

High-Z is detected in the following way:

  1. Drive the line low
  2. Convert to input (high-Z) and read
  3. Drive the line high
  4. Convert to input (high-Z) and read

If the two input readings hold different values, the pin is in high-Z state.

Use an ADC to Encode Version

Another method for supporting version numbers is to use an ADC to convert a voltage to a version. A resistor network is used to control the voltage seen at the ADC (the midpoint of the resistor network is connected to the ADC input). Versions correspond to “binned” ADC readings, representing equal divisions of the full ADC range (e.g., 4095 for a 12-bit ADC).

Version ADC Range (12-bit)
1 0-511
2 512-1023
3 1024-1535
4 1536-2047
5 2048-2559
6 2560-3071
7 3072-3583
8 3584-4095

Values for the resistor network should be chosen so that the target voltage/reading falls within the midpoint of the range specified. This scheme also necessitates using high-precision components (1%) that will not de-rate over the course of the product’s lifetime such that the version falls outside of the specified range.

Resistor values need to be appropriately selected to minimize leakage current. In addition, a second GPIO can be assigned to the versioning scheme. This pin will supply VCC to the resistor network, and it can be turned on only when the software needs to read the board version.

Note

A larger range of versions can be supported by tightening the binning on each version. However, tightening bins too much may lead to de-rated resistors causing erroneous version reads later in the product’s lifetime.

You can also combine the ADC and GPIO approaches, using one GPIO pin + one ADC pin. This would double the number of values supported by the ADC approach.

Write Hardware Version in Persistent Memory

Some processors provide user-configurable non-volatile memory. These registers can be written during manufacturing, storing the associated hardware revision directly in processor memory. You can also use a dedicated configuration EEPROM or a reserved region of flash to store this information.

Ideally, these registers/regions could be locked so that the version (and any other configuration information) cannot be accidentally overwritten.

Prototyping New Revisions

With software-readable board versioning, it becomes possible for software to be updated and tested prior to having new hardware produced. Existing boards can be reworked to match the new version as closely as possible, and that board can have its resistor network changed to the next revision. Software developers can use this reworked hardware to add support for the new revision in software.

Note

Not all changes are easily supported via rework, so some changes will need to land once the final hardware is ready. Nonetheless, we have used this strategy on multiple occasions to make significant progress ahead of receiving new hardware.

“Securing” Hardware Revision

Common practice involves stuffing resistors on traces present on the PCB, which enables easier rework. However, for teams that are concerned about CM’s “cheating” by marking old PCBs with a new revision ID, you can hard-code the board revision using the GPIO method by laying it out within the PCB (i.e., wiring the traces internally to encode the 0’s and 1’s of the PVB layout. We do not recommend this during development, as the rework method above is often quite useful, but it is suitable for production.

Further Reading

Part Numbering Scheme

14 January 2022 by Phillip Johnston • Last updated 14 July 2022We like a simple part numbering scheme, something like: XXX-YYYY-RR (with or without dashes). XXX: Class/Category prefixes These are handy for quick recognition but especially for storage YYYY: Simple incrementing numerical ID RR: Revision Useful Rules of Thumb All part numbers should be the same length No leading zeroes – people will be tempted to truncate the leading digits from the number. (000123 → 123) Numbers are faster to input than letters, and the numeric keypad allows single-hand operation. For this reason, you may consider not using letters in …

To access this content, you must purchase a Membership - check out the different options here. If you're a member, log in.

Source Control Commit Guidelines

Preparing quality commits in a source control system is an underrated skill. Commit messages help explain the context for set of changes. We can’t always find an explanation or context for why code was changed by looking at the code. We must keep a detailed project history in some other way, and the commit tree is one of the best places. This history is invaluable when debugging future problems or trying to understand why a previous change was made.

Additionally, how we structure our commits matters. With commits that are too large, reviewers will have trouble processing all of the information. Future debugging efforts are also hindered, because developers cannot pinpoint exactly where an error was introduced when large commits are the norm. We want to build a project history that is truly a future resource for future developers.

The following commit guidelines are used on Embedded Artistry open source software projects to provide a consistent organization style. If your commits and messages are not of sufficient quality, you may be asked to amend them before a pull request is accepted.

In general, these guidelines will serve you well no matter your organization. For other takes on writing quality commit messages, see the Further Reading section below.

Table of Contents:

  1. Prefer Atomic Commits
    1. Splitting Commits
    2. Whitespace and Formatting Cleanup
  2. Commit Message Guidelines
    1. The Subject
    2. The Message Body
    3. Reference Issue Numbers
    4. On Cherry-Picked Commits
  3. Revise Your Commits Before Submitting a PR
  4. Further Reading

Prefer Atomic Commits

In general, the smaller the commit, the better. We prefer atomic commits, which apply a distinct and related set of changes in a single action.

What does this mean in practical terms?

  1. All changes in a commit should be related.
    1. Don’t combine changes that address different problems into a single commit.
  2. All changes in a commit should address a single aspect of a problem or change.
    1. Don’t be afraid to break up a new feature, bug fix, or refactoring effort into multiple distinct changes.
    2. This will help you keep track of what works and what doesn’t: if there’s a problem, you can always revert back to the last known good state. If you wait too long between commits, you may lose a lot of work or spend too long finding the source of the problem.
  3. Prefer small commits to large commits.
    1. This helps reviewers by allowing them to focus on a small set of related changes.
    2. This helps future debugging efforts by increasing the probability that a git bisect operation will quickly identify the source of the problem.

Splitting Commits

Sometimes, we accidentally commit distinct sets of changes together. Or we might mess up a rebase effort, accidentally squashing two unrelated commits.

To undo this problem, review our process on splitting up git commits in the middle of a branch.

Whitespace and Formatting Cleanup

Don’t mix code changes with whitespace or formatting changes! If you are fixing code formatting, include those changes separately from your code changes. If your request is unreadable due to whitespace changes, it will be rejected.

Many of our projects enforce automated code formatting using clang-format. Formatting is enforced by the Jenkins build server which runs continuous integration for this project. Your Pull Request cannot be accepted if the formatting check fails. If your changes are rejected by the server, please include the auto-format updates in a separate commit.

You can auto-format your code to match the style guidelines by issuing the following command:

make format

If you don’t have clang-format installed, we can apply the format changes for you.

Commit Message Guidelines

Below is a visual representation of the Embedded Artistry git commit message format:

50-char-ish"subject" which summarizes the change

After the first line, include additional explanatory text. Include
one blank line in between the summary and the body. Various tools
like git log, git shortlog, and git rebase get confused if the two
sections run together.

You can also include bullet points:
* A
* B
* C

Reference issues at the bottom, like this:

Fixes #102
Relates to #100, #8

The Subject

Treat the first line of a commit message like an email subject line.

We recommend a maximum length of 50 characters because it works best with git’s command line tools.

We also recommend using the imperative mood for subjects. This might be what you think of as “commanding” or “instructing”. Examples include:

  • Refactor subsystem X
  • Implement feature Y
  • Remove deprecated methods
  • Update version to 1.2.3

Another way to look at it is to have your subject line complete the following sentence:

If applied, this commit will [subject line here].

We don’t worry about manual line breaks in our commit message bodies. Text wrapping works well in modern tools.

The Message Body

Please separate the body from the subject with a blank line.

Most importantly, the commit message body should be used to explain what you are doing and why you did it that way, rather than how you did it. The code itself serves to explain the how. Focus on side effects, compatibility changes, or other consequences that are not immediately obvious from reviewing the code. Also include any important factors that helped you arrive at your particular approach.

Not all commit messages require both a subject and a body. You can include only a subject if it is sufficient for a given commit.

Reference Issue Numbers

When working with GitHub projects, reference relevant issue numbers in your commit message body.

You can have a commit automatically close an issue when it is merged to master by saying something like the following in your message body:

Fixes #123
Fixes: #123
Resolves: #123

You can also reference issues in other repositories by adding a prefix for organization/repository:

Fixes octo-org/octo-repo#100
Further reading

The full list of keywords that will close an issue is found in the GitHub documentation.

If other issues or PRs are related to this commit, but it doesn’t address the problem, you can include the issue/PR number without using a keyword like “fixes” or “resolves”. This will link the commit and the issue in GitHub’s web interface.

On Cherry-Picked Commits

Sometimes, we cherry-pick changes from one branch to another. We recommend using the -x flag with git cherry-pick, which references the original commit ID.

For more information, see Git Cherry-pick Recommendations.

Revise Your Commits Before Submitting a PR

While you’re working on a problem on your own branch, don’t worry about getting these commit details perfect. Instead, clean up your branch before you submit a PR. You can perform a git rebase to squash commits, amend commit messages, and more.

We often use a visual tool, such as Sourcetree, for complex rebase process.

Further Reading

For more on writing quality git commit messages:

For more valuable processes to follow: