We came across a question on the Embedded.fm Slack Group that we wanted to tackle:
Are there any good resources out there for designing forward & backward compatibility into a new system? I manage a fair amount of testing suites and as I’ve upgraded or added new features, I’ve inadvertently broken scripts from previous releases that expect a particular interface to invoke the tests properly. A command line argument becomes obsolete, a new function needs to be called that didn’t exist before, etc.
We definitely understand this type of pain! We hope the strategies outlined below can help you manage compatibility your own programs. We’d also love to hear your approach to managing backward compatibility – tell us about it in the comments!
Table of Contents:
Strategies for Managing Backward Compatibility
A rough rule of thumb for maintaining backward compatibility is to never remove anything from or change anything about an interface. Of course, we rarely get things right on the first try. Even if we did, the environment will eventually change enough such that we must adapt.
The following high-level strategies can help you manage API compatibility as your system evolves:
- Use Explicit Versioning
- Add Dynamic Capability Discovery
- Add New Optional Parameters or New Methods
- Introduce New Parameters or New Data Types in New Methods
- Create Wrappers
Use Explicit Versioning
Explicit versioning can help your users handle incompatible changes when they have to be made The way to do this is to support both the old and new API versions simultaneously. The older version will be marked as deprecated, but your users will not have their programs break immediately.
We often see this with web interfaces, where the API version is encoded in the URL: website.com/api/v2/endpoint/
Multiple versions are live simultaneously and only removed following the deprecation process.
You might use namespaces to control the version if your language supports it. One example to consider is the C++ STL’s std vs std2 namespaces.
Add Dynamic Capability Discovery
Some changes can be managed through dynamic capability discovery, where you provide some method that can be used to query the supported capabilities of an application or endpoint. This might be through a get_info or has_capability or supports function that uses an enum or other identifier to account for a given capability set. The endpoint can respond with a success or error code that indicates whether the capability can be used.
You can also query a version number which can be mapped to a capability set internally.
In either case, the goal is for an application to proactively manage capability mismatches based on what the endpoint reports.
Add New Optional Parameters or New Methods
Some languages support optional parameters that have a default value. If the parameter is not supplied, the default is used. This enables us to maintain source compatibility for programs using an older method invocation. If you need to expand a function’s parameters, add optional parameters instead of required ones.
Note: For IDLs, such as Thrift, you will often encounter the advice: make everything optional except for fields that cannot ever be empty or removed. This advice works well for RPC interfaces, which deal with serialization/deserialization and transmitting data over the wire. Making fields optional allows you to easily remove them at a future time if necessary.
Some languages, such as C, do not support optional parameters. In other languages, adding optional parameters or function overloads can introduce an incompatible change (e.g., in C++, a cast may be required for an overload). In these cases, you will need to create a new method instead of adding an optional parameter. You may also want to introduce a new API due to your use of explicit versioning.
When creating a new method, use a distinct name or suffix the function with a version identifier:
int64_t function(int32_t a, int32_t b);
int64_t function_v2(int32_t a, int32_t b, int32_t c);Where to draw the line between adding a new parameter and adding a new method is somewhat subjective. For small changes, such as changing a mode used under the hood, an optional modevariable with a default value will often suffice. Larger changes often lend themselves to creating new methods. If you wish to deprecate the older method and exclusively use the newer method, you will want to split the functions apart so you can begin the deprecation process for the outdated function.
Introduce New Parameters or New Data Types in New Methods
Another reason to introduce a new method is to avoid data incompatibility. Create a new method whenever you need to introduce a new data type, introduce a new required parameter, or modify existing types.
When creating a new method, use a distinct name or suffix the function with a version identifier:
int64_t function(int32_t a, int32_t b);
int64_t function_v2(int32_t a, int32_t b, overflow_t c);If you wish to deprecate the older method and exclusively use the newer method, you will want to split the functions apart so you can begin the deprecation process for the outdated function.
Create Wrappers
Sometimes we have to make changes to types used internally on our system. However, these types do not always need to be exposed over existing remote interfaces. One data type can often be mapped to another data type that is expected by the internal APIs.
If possible, avoid modifying user-facing APIs to reflect internal type changes. Instead, implement a new method using the new types. Keep the old method around, but translate the input types into those expected by the new method.
Define a Deprecation & Breaking Change Policy
Systems and APIs evolve over time, and we must adapt accordingly. Regardless of the efforts you take in maintaining backward compatibility, there will eventually be an occasion where you must break with the past. We recommend setting a public deprecation policy for your systems so that users know what to expect when breaking changes occur.
We recommend providing your users with a chance to migrate their systems to the new APIs before completely breaking their programs. You will often encounter “two release” deprecation cycles, where there is at least one release which provides a deprecation warning before the breaking change is made. Some systems, such as RESTful servers, will often keep old APIs active for a specific timeframe, allowing users to migrate to the newer interface at their own pace. It is up to you and your users to agree on an acceptable timeline for deprecation.
Here is an example policy:
- The behavior of an API must not change between any two consecutive releases unless it is going through the deprecation process
- An API cannot be removed without notice between any two consecutive releases
- When a breaking change is required, the following process will be observed:
- The planned change and overall timeline will be communicated to the users via any number of methods:
- Buck tracker
- Email list
- Slack Group
- Release notes
- Social media
- Users of the existing API will be given a chance to comment and provide feedback
- Release:
- Existing APIs still function properly without breaking
- You can still add new methods or parameters support the new functionality, as long as the old functionality continues to work
- Add a deprecation warning if possible
- Announce start of deprecation period in your release notes
- Existing APIs still function properly without breaking
- Release + 1:
- Remove deprecated items
- Make breaking changes
- Announce breaking changes and end of deprecation period in release notes
- The planned change and overall timeline will be communicated to the users via any number of methods:
More for Members
Our members can find additional information, strategies, and rules of thumb in the Embedded Systems Field Atlas. Check out these entries: