Q&A: On Managing External Dependencies

22 June 2020 by Phillip Johnston • Last updated 2 January 2024

One of our clients approached us with questions about managing external source code:

We’ll address these questions in sections:

1. To Fork or Not to Fork
2. Copying Files into a Repository
3. Dependency Management Tools
4. Monorepo Strategy
   1. A Middle Ground for Monorepo Setups
   2. Migrating to a Monorepo
5. Our Approach
6. Further Reading

To Fork or Not to Fork

Our own tendency is to rely on the primary repository if we’re using it directly and not making any changes. This allows us to easily stay up-to-date with the parent project, since we can just pull updates as needed.

We typically fork repositories when we are going to modify the project for our own purposes, or whenever we need to submit changes to the projects via a pull request. Forking so that an organization maintains a copy of the code also makes sense, and we’ve worked with many teams who do just that. Projects can certainly disappear, but luckily we have yet to experience that for dependencies.

The tradeoff with having a fork of a project is that it requires additional work to keep your fork in sync with the upstream repository. You have to manually add the original project as a new remote, pull in the changes from the remote, and merge them into your repository.

Copying Files into a Repository

Some people don’t want to deal with forks or submodules. If you still want to maintain a copy of external code in your projects, you can follow this approach. We recommend creating a script to automate this process.

• Clone the project or download zip archive of the source code using wget, Git, or another tool
  • Ideally, you will store the files in an archive of some type (e.g., .zip) – so you may need to zip the source files after cloning them with Git
• Store the source archives in a standard location in your project tree
  • This is useful because you maintain an original copy of the source, then you can always get back to a fresh state when you need to
• Unzip the archives and place the files in their desired locations in your source tree
• After you’ve unzipped the source, apply patch files containing changes to the original source code
  • If you need to modify any of the files for any reason, keep a set of patch files under source control
  • This is a much more sustainable way to separate your changes from the primary codes and provides for easier updates
  • Eventually, given enough changes in the primary code, the patches will need to be updated – but that’s easier than manually re-applying updates every time

This script can be manually run so you control the update frequency. You could also tie the script downloads to specific releases, giving you fine-grain control over updates.

Dependency Management Tools

Rather than manually copying the source for an external project into a repository or including the external project as a submodule, you could also use a dependency management tool.

Such tools include package managers like Homebrew, apt, and Python’s pip. This also includes external source code dependency management tools like Conan, Vcpkg, or the CMake CPM module.

Rather than storing dependencies inside of our source tree, we can rely on these dependency management tools to download dependencies. We can pin dependencies to specific versions, or get the latest version. These tools can also be configured to point to our project forks if needed.

These tools still leave you with the potential risk of a project disappearing on you. However, every dependency management tool that we rely on allows you to download to and install from a local cache. Many dependency management tools will even maintain a local cache of package installers and tarballs by default.

For teams that are worried about disappearing dependencies, we typically recommend configuring the company’s build server to cache dependencies. This is often done with a combination of tools, such as creating an apt-cache server to cache installable packages, and relying on local Conan and CPM caching for source dependencies.

You can create a CI process step that performs a backup of the dependency caches, ensuring you have multiple snapshots that you can revert to in case you need an older version that is no longer publicly available. You could also create a CI job that copies dependency packages to a location on the build server which is accessible to your team in the event that they are unable to download a dependency.

Monorepo Strategy

With regards to storing code in a single repository or in separate repositories, there are tradeoffs to both approaches.

The primary benefits that we’ve seen with a monorepo approach are improved code sharing and simplified versioning. This is increasingly relevant for modern embedded projects, which often involve multiple integrated software systems working together. For example, we’ve worked on a system that involved an iPhone app, an Android app, an embedded Linux computer, a microcontroller-based embedded device, and a web server.

In that complex system, all of the devices communicated with each other, and thus the systems needed to share the definitions & types for the communication interface. We started by having communication definitions in a separate repository. This repository was included as a submodule in other projects. Things quickly got out of sync as updates frequently landed in one system before another, breaking compatibility. We also needed to maintain a matrix of version numbers for each release, as each system was versioned differently.

We moved to one repository that contained all of the code for the different systems. If you updated communication code, you immediately knew if you broke something in another system (from the CI server at least, if not from testing on your local machine). In this setup, cross-system changes and updates were made on a single branch and were merged together: no extra coordination needed. We also had uniform versioning across each system because we used a single repository. There was no longer a need to keep a matrix of versions for each release.

Having just described the obvious advantages of a monorepo setup, there are also tradeoffs to consider.

With extremely large code bases, storing everything in a single project can become a problem with certain source control systems. For example, with Git there is no ability to checkout only a portion of the tree; you can only checkout a portion of the Git history. Other tools, such as subversion and Perforce, allow you to checkout only the pieces of the project that you’re interested in. At a small scale, this is not a problem. At larger scales (e.g., Apple and Google), you would need to migrate to a non-Git system to sanely manage everything in a single repository.

Another disadvantage is that a pure monorepo setup does hinder sharing subsets of the code outside of a project. This may not matter at all for your organization’s needs. Subsets that need to be shared can always be split off into submodules with Git.

A Middle Ground for Monorepo Setups

There is a middle ground between a pure monorepo setup and a fully distributed project setup: building a monorepo out of submodules or by downloading dependencies (i.e., using a dependency management tool).

The “monorepo” in this case is responsible for coordinating the overall build and versioning processes across the entire system. Components of the system may be able to be built and managed individually, but the CI process will detect any potential interface mismatches and incompatible versions.

The important thing to keep in mind is that you don’t need to be a monorepo purist. External libraries and subsets of your system that you want to be shared externally can always be included as submodules or external dependencies. You still maintain the benefits of sharing code and simplified versioning, even if you have some additional overhead in managing and updating dependencies.

Migrating to a Monorepo

This client responded by asking for advice on migrating to a monorepo strategy in incremental steps.

Here’s our recommended approach for git:

• Create a new repository to house the monorepo setup
• Integrate the existing projects as submodules
  • Recreate build rules in the top-level project, starting to organize as needed (e.g., libraries/, ios/)
  • The goal is to make sure everything works in the monorepo setup before you fully commit
• Start removing duplication from the projects, e.g. shared libraries should be referenced just once in a monorepo setup, instead of each subproject containing its own copy/reference
  • This can be done within branches of the submodules, allowing the original projects to remain useable during the migration period
• If you want to kill the submodules eventually, start pulling in projects and adjusting them one-at-a-time, until all of the code is eventually in your single repository
• Add setup scripts, installers, other shared resources to the monorepo in a consistent location (tools/, scripts/)

Our Approach

There are many options available for managing your team’s external dependencies. The right solution ultimately depends on the complexity of the project, the actual risk of dependencies disappearing, the cost to the project if critical dependencies disappear, and the team’s overall paranoia level.

We do not worry too much about our dependencies disappearing, which drives our overall process.

• We include external source code in our projects with dependency management tools
  • When we cannot rely on a dependency management tool, we’ll use submodules
• We fork repositories only when we want to make a modification to the upstream project
  • If we can get our changes merged into the upstream project, we delete our fork
• Our build server is configured to cache dependencies
• Critical installers are stored on the build server
• We manage most of our projects as standalone repositories, especially libraries that are shared across projects
• Whenever we have a complex system involving multiple subsystems that work together, we prefer using a monorepo strategy
  • If we start the project knowing this, the project will be managed inside of a monorepo with external dependencies managed as submodules or by a dependency management tool
  • If we start the project with separate repositories, we’ll create a “psueudo-monorepo” that includes the various projects as dependencies and coordinates the build and versioning process.

Further Reading

• Embedded Artistry’s GitHub Process
• Embedded Artistry’s Continuous Integration Process
• Why Google Stores Billions of Lines of Code in a Single Repository
• Monorepo. One Repo To Rule Them All.
• Apt-cacher-server
• Setting up an apt Package Cache
• Mastering Conan – Conan local cache: concurrency, CI, isolation
• TheLartians/CPM.cmake
• Monorepo Development
• An Experiment: Develop in a Monorepo and Distribute to Standalone Repositories