Upgrade:

We have two endpoints/ports, one for gRPC and one for REST. Both of these need to implement versioning such that we are able to handle both backward and forward compatibility.

While handling versioning, we should also show prudence while making updates to our api definitions to be backward compatible and avoid changes that can make the api backward incompatible.

Backwards-compatibility:

• We can add new methods to API interface
• We can add new fields to a request message
• We can add new fields to a response message
• We should not remove or rename an API or field
• We should not change type of a field
• We should not change core behavior of existing request processing

Forward compatibility:

• API implementation on service should be able to handle/ignore extra fields in the request
• Clients should be able to handle response with lesser fields
• Clients should be able to handle Method-Not-Found class of Exceptions and fall back on older behavior

REST:

We already have a version added to our uri path.

/v1/resource/entity

Clients can request resource based on the version and if server supports the version, the server processes the request and sends the response back. Else controller responds with appropriate HTTP error code.

Between releases we can have minor version updates to the API versioning and we can follow Semantic Versioning approach.

We can also have a getVersion api which lets the client know of the latest version supported by the server.

gRPC:

The protobuf serialization takes care of forward and backward compatibility. At the protobuf level, we can add new fields and never reuse a field number to ensure our backward and forward compatibility across revisions are taken care of. For different versions, when we need to significantly change the message format for request or response, we will compose messages such that we carry both older and newer versions.

For example:

message MyMessage {
 MyMessageV1 mv1;
 MyMessageV2 mv2;
}

We will also ensure that we never remove/replace APIs to keep our service backward compatible.

The two interesting cases that we need to describe in detail are 1) new RPC methods added 2) RPC method signature altered.

1. Altering signature of apis can be made both backward and forward compatible by following a scheme where client sends multiple versions in the request and server responds with the highest version it supports.

rpc MyApi(MyRequest m) returns MyResponse

Messages with version and revision. MyRequestVi has multiple revisions that stay compatible.

message MyRequest {
 MyRequestV1 mv1;
 MyRequestV2 mv2;
}

message MyResponse {
 MyResponseV1 mv1;
 MyResponseV2 mv2;
}

1. Addition of new RPC methods has no effect on backward compatibility but breaks the forward compatibility where client has a higher version than server and makes rpc method call that is not defined on the server. The server should respond with appropriate error code to let the client know to let the client know the api is not supported.

Metadata:

We follow our custom serialization that supports versions and revisions. Revisions are backward and forward compatible while versions are not. It's a trivial matter to upgrade controller instances for a revision upgrade to metadata. Each controller instance during the upgrade will be able to work with (both read and write) older and newer revisions of data.

However, with a version upgrade of metadata, we need to handle the upgrade path differently so that the service stays available as we upgrade different controller instances without breaking forward compatibility.

Upgrade Strategy: So we will have to follow an approach where we first upgrade the binary on all controller instances enabling them to handle higher versions while ensuring that we do not perform any writes using the higher version yet. This will make all controller instances capable, of reading new versions of data while they continue using older version for writes. Once all controller instances have been upgraded, we can change the configuration to allow them to write newer versions too. This approach will ensure that each controller instance is always able to handle all the stored metadata even though newer version breaks the forward compatibility.

Rollbacks with versions and revisions: There is a still a problem if rollbacks were to happen with forward compatibility and it should be handled very carefully. We don't have a neat solution for forward compatibility in case data is written with incompatible newer version while controllers are rolled back to lower version.

So for metadata, we should ideally try not to have incompatible versions of metadata, rather only use revisions. However, if a need for adding breaking version arises, and there is a case for rolling it back, we should have tooling to transform metadata from newer versions into older versions before we initiate rollbacks. It may require service downtime.

Alternatively we could only allow roll backs across revisions but never across versions.

Metadata Tables:

Another aspect of controller metadata is handling of tables. We do not want to serialize and deserialize the entire table everytime we read it or want to append to it. We would still want to retain append only properties of our tables. This means each row within our tables could now have different schemas.

This means, for our tables, the assumption around fixed sized segment rows can no longer be assumed across different versions. For this we will need to maintain a separate index for each table which identifies starting offset of each row. So it will be simple <row-key, offset-in-table>. This will allow us to maintain different schemas across rows in the table while continuing to be as efficient wrt query performance.

We will also add bidirectional row delimeters at the start and end of each row, with lengths/pointers to next and previous entries. That will enable us to continue to can read any arbitrary entry in the table and traverse in either direction without having to deserialize all the entries in the table.