API Evolution

Convener

Joe Betz (@jpbetz)

Attendees

Dave Howell, Rob Raux, Adam Gainsley, Matthew Campo, Petre Popescu, Jerome Louvel, John Fair, John Held, Mitch Oliver, Todd Fredrich, Filip Kolarik, ??

Notes

Attendees briefly described how they currently manage API evolution, what they plan to do in the future and any issues they are currently facing. The main take aways from this exercise were:

• Many use a version number in the URL, e.g.: http://domain/v1/ style for backward incompatible versioning. This was brought into question, see below.
• Some are using or considering using Content Negotiation. E.g. a client includes a Accept header with "application/json;version=1" when requesting a representation to get a version of a resource. A longer discussion on this is detailed below.
• A few have successfully used hypermedia and Loose Coupling to eliminate the need for API and resource level versioning and are considering using Content Negotiation for representation level versioning.
• A significant number of participants have a mixed environment of various RPC frameworks and REST and they're looking for processes that will ease the migration from RPC to REST.
• Some API upgrades (SOAP was mentioned) had to be done in lockstep with client upgrades. The described upgrade approach was to bring the server and clients out of rotation (they cannot accept traffic during the upgrade), upgrade all of them, and put them back in rotation. If any of the clients or the server has a defect, they must all be rolled back. This versioning strategy has negative impact on development and development methodology, is unscalable, and would simply not work in an open ecosystem.

Questions asked by Attendees during exercise:

• How to migrate from non-REST to REST?
• What Migration/Deprecation Strategies work? how to notify client?
  • how to incentivize clients to upgrade to a later version. Some migrations are to a newer version of a REST API, others are migrations from non-REST to REST.
• How does one manage lifetime for a version so that it does not need to be maintained forever?
  • One major restriction is contracts.
  • Knowing the cost to serve is being used in some ecosystems to have the client either foot the bill to keep the old version running.

Major Discussion Topics

While a theme in the discussion was how to evolve with minimal versioning, the major levels of versioning discussed were:

• API level - E.g. http://domain/v1/ style. The idea is that the version applies to the entire API, including all resources and representations. This approach was brought into question, see below.
• resource level - There was a lot of valuable feedback on how versioning at the resource level can be avoided. However, use cases and patterns where backward incompatible changes to resource directly (i.e. not to the representation but to that actual resource, such as what methods are supported or what query params a method takes) were not discussed.
• representation - There was a general consensus that this is where most changes happen and where versioning is most needed. Content negation was discussed as a mechanism to enable.
• behavioral - e.g. core business logic has changed, this impacts not only the syntax of the API but also the expected behavior. This was also not covered in details, there were suggestions that this can usually be done in a backward compatible way.
• Not related to API evolution but mentioned was instance data versioning (e.g. versions of a wiki page, often using optimistic concurrency control).

Concerns with using /v1/ to scope all resources under a version:

• For apis with multiple resources, a client may want to selectively upgrade from v1 to v2. In the presence of hypermedia, this would required all links be to the same version path that the resource they were requested from is in. But to get around clients have been known to hack the links (e.g. regex s/v1/v2) so that they can use v2 of the resources they have upgraded to and v1 for those they have not yet upgraded to. This highlights how violating the uniform identification property of URIs (there are now two URI representing the resource, one with /v1/ and another with /v2/) can be problematic.
• It was pointed out that often if a incompatible idea of just a resource, not it's representation, needs to be made, this may justify creating a new resource rather than attempting to version the entire API. Also, version numbers in URIs are not necessarily needed for representation versioning, which can be handled using content negotiation.

Details of using Content Negotiation to version representations:

• Only need a new version when introducing backward-incompatible changes, all backward compatible changes can be made to the same version
• Client sends Accept header with "application/json;version=1", server responds with content in that version of representation OR a 406 Not Acceptable if the resource does not support that version
• Discussed how a server can respond when a version number is not provided. Options: (1) Respond with oldest supported version, (2) Respond with newest supported version (3) Respond with 406 Not Acceptable and short human readable body explaining that version # is required. #3 is considered the most conservative, with #1 also being fairly conservative if old versions are supported indefinitely but just as risky as #3 if server deprecates versions over time. #3 is the most risky as a client may be unable to interpret a new version of a resource.
• Discussed level of granularity of representation versions.

Visibility into how clients use APIs:

• Metrics on what APIs are being called by what clients
• Eg. what resources they call, what methods they use, even what fields they access (can be gathered if clients use projections) are a core asset when evolving APIs. It allows the provider to determine what clients may be impacted.

Rate of change is high during early development:

• Eat your own dog food.
• Keep internal long enough for the API to stabilize and mature.
• Consider initially providing the API as a "closed beta". Only select clients may use API, API is allowed to go through backward incompatible change until it's out of beta. Client should not use in production until out of beta.

Outcomes

• Lockstep is obviously horrible! This is the "anti-pattern" for API evolution. A tolerance for backward compatible changes (e.g. adding optional fields, new resource methods) is a must.
• Detailed Metrics on how clients call your API is key in making informed API evolution decisions, as well as for generally staying in touch with your clients needs and expectations.
• Content Negotiation for representation level versioning (of backward incompatible changes) has promise, not all details have been worked out.
• Placing all resources under a /v1/ was brought into question, strong arguments were made that this approach, while common, both breaks uniform identification properties of URIs and also can cause some non-obvious problem down the road. See above.

Outstanding Issues

• Content Negotiation for representation level versioning (of backward incompatible changes) has promise, not all details have been worked out. References or write ups of working strategies needed.
• The largest practical concern is how to ease migrations from non-REST to REST APIs. This was not answered.