It’s time for a blanket statement: users will prefer improving extending existing API versions over adopting a new version due to a backwards-incompatible change.

The reason: the cost to the user to start using the next version of the API is larger than any value they would derive from that next version alone (e.g. sans new features).

Let’s break this down.

Definitions #

For the context of this article, I wanted to scope the conversation with the following terms:

  • API: I’m really talking about remote APIs: any programming interface that is performing a remote procedure call. The simplest example is HTTP + JSON, so I’ll be using that for the rest of this article.

What types of changes require a new API version #

Generally speaking, a new API version should only be published in the case where the API author needs to introduce a backwards-incompatible change (e.g. a semver-like convention). The scope of a backwards-incompatible change varies depending on the context, but examples include:

  • Renaming a field.
  • Modifying the schema of an object.
  • Introducing entirely new conventions and handling of values.

The motivation for a new API when introducing breaking changes is reasonable: API authors should build trust with their users by providing a mechanism to notify them when there is a need for them to change their code to interface with said API. Versioning makes the choice on the user explicit to upgrade.

But API migration is expensive for customers #

The challenge comes in the logistics of the users upgrading their clients.

Often API authors can find themselves sufficiently removed from the problem of dealing with the churn of API upgrades, and get an unrealistic idea of the cost of the API migration.

Some of the reasons upgrades can become expensive are outlined below.

Tightly coupled dependencies requiring lockstep upgrades #

Clients and SDKs to interface with APIs are often not written in a highly modular fashion: instead, they are whole surfaces that must be upgraded, such as with all the APIs on a cloud like AWS or GCP, or for all services offered by a platform like Stripe.

The breadth of this surface often means that you update your code for multiple different clients at once, often unrelated to the one service or tool that you want to use.

Upgrades can be multiple layers deep #

In some cases, although the upgrade of the client is trivial, it can be nested a dependency chain that makes upgrades take significantly more effort and time.

For example, consider the usage of a raw SDK, wrapped in a convenience wrapper like a Terraform provider. The sketch could look like:

  • Raw API.
  • Go SDK.
  • Terraform provider.
Command Not Found: mmdc

In order for the user to finally get the upgraded SDK, they have to update the Go SDK version, and update the Terraform provider it is chained to. Each have their own cost associated, and an incompatible change can leak the cost of the upgrade into every downstream.

Not only does this make upgrades more costly, but it can make them take longer: what if the team who upgrades the provider is separate from the team who upgrades the SDK? What if one of those pieces is an open source project that is swamped and slow to respond? This can make a simple code change extend into a multi-week or sometimes multi-month endeavor, including the cost of coordination overhead.

API upgrades do not benefit the end user #

We’ve established that an API upgrade can be expensive. What about the benefits?

The reality is the reasons that engineers often want to introduce a new change isn’t particularly valuable for the end user. Let’s look at some of these arguments now.

API intuitiveness #

The most common example of improving usability is renaming a field in an API payload: the previous name didn’t really capture the purpose, so the field should be renamed to something that does.

There’s similar changes in the category as well:

  • Moving fields from one object to another.
  • Gathering a set of fields and interning that into a subobject.

The problem is intuitiveness of anything is highly subjective: intuition around a user interface is dependent on the experience that an individual had previously, which itself is dependent on the frequency of an accepted idiom.

Consider the now-famous hamburger icon (≡) or triple dots (⋮) that we see on every website: neither was “intuitive” until they become common in styles guides and applications. Someone viewing these for the first type wouldn’t immediately be able to reason what they mean: they have to be taught.

Therefore, one has to assume that, unless a user can somehow completely intuit the schema of an API, they will be forced to look up some form of documentation to be able to use the API.

Even if a user could theoretically intuit the whole API, they will still be likely to look it up, and have to verify the meaning of every field: if you had the choice of just looking up what these terms mean or a workflow of guess-and-verify-the-field-does-what-I-think-it-does, many would look up the meaning to save the time spent fiddling with the API if they’re wrong.

Therefore, API intuitiveness is largely irrelevant: accurate, clear documentation or examples will be more valuable every time, and comes with zero cost in end-user toil to update their API calls to new schemas or field names.

Most API changes can live on the same version #

Most API changes can actually live on the same version! In some of the examples above, instead of introducing a new version of an API entirely, one could instead:

  1. Introduce a new field with the new name / schema / behavior.
  2. Accept both indefinitely, and add validation to make new fields mutually exclusively with any old conflicting fields.

This is largely similar to the burden of maintaining multiple version of the API simultaneously, but comes with the benefit that the work is largely additive: the work to use a new feature is only the cost of adding support for that field in one’s SDK / library, and is not coupled with other additional burdens that don’t provide immediate value (like refactoring your integration to support some new v2 schema).

New Features #

Often new APIs versions also become the only way to consume new features in the underlying service: engineers don’t want to have to maintain multiple code paths or update legacy ones to support new features, so new fields will only be introduced in the new API.

However, there is no technical reason why these fields cannot be introduced in the older API versions, and this in turn results additional user friction: the user is made to pay the cost of an expensive upgrade, even to get a single feature flag.

Examples around the internet of pain of migration costs #

I think there’s very few tangible examples of someone praising a new API, but there’s a plentiful amount of complaints around an API changing and the users being unhappy with paying the cost to upgrade:

Summary #

  • A new version of an API is expensive for consumers to migrate to.
    • Often clients have a much wider surface area than the single API being upgraded, and therefore are that much more expensive to upgrade.
    • Clients and SDKs tend to have a chain of dependencies (e.g. wrapped providers or CLIs wrapping SDKs), resulting in significant coordination to support new APIs.
  • Many of the percieved benefits of new APIs aren’t true in practice
    • API changes that are more “intuitive” is highly subjective, and without sufficient standardization will require the user to look up documentation. Documentation is the default fallback and the first things users look at, regardless of schema of the payload.
    • New features and fields could be added to the old API without issue.
    • Many desired changes can be made incrementally to existing API version, by introducing new fields with mutual exclusivity.