Protobuf API Contract Guideline

This post provides guidelines to manage APIs using protocol buffers. Check out this blog post to learn more about Protobuf API contract management.

API Contracts

API contracts describe the surface area of the request and response of each individual API method being offered. It is something that both API providers and API consumers can agree upon and get to work developing and delivering, and then integrating and consuming. An API contract is a shared understanding of what the capabilities of a digital interface are, allowing for applications to be programmed on top of.

Google Protocol Buffers: Protobuf

From the Google Protocol Buffer website:

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Using Buf

When working with protobufs, Buf's linter is the tool to use for checking the quality of protobufs.

Use Buf to automate adherence to an agreed upon, community-tried-and-tested style guide.

Use this linter to check that:

• File names adhere to the naming convention.

• Files with package X must be within directory X relative to the root.

• Service names end in Service.

• Method names are PascalCase.

• Field names are lower_snake_case.

• Fields and messages have a non-empty comment for documentation.

• Enumerations have a proper zero-value default and enum values are properly named (with a prefix).

In addition to the above checks, Buf automates the detection of breaking changes (i.e. making new interfaces incompatible with existing clients) and reduces the time required to manage and code review proto files, as part of the code review is automated.

Using the DEFAULT Lint Category

There’s no reason to not use the DEFAULT lint category. This is the most “strict” category that encompasses MINIMAL and BASIC categories. A description of the lint categories and styles enforced by the DEFAULT category can be found in the Checkers and Categories section of the Buf online documentation.

In addition, to ensure that all message types and elements are documented, the COMMENTS category must also be included.

In the buf.yaml file, the two categories are enabled like thi:

 lint:

   use:

     - DEFAULT

     - COMMENTS

The DEFAULT category influences the recommended stylistic approach documented below.

Unique Messages for RPC Requests and Responses

It’s important to keep in mind that APIs may change over time, and you probably don’t want to couple two separate RPC calls tightly together.

Exercise Caution When Creating Generic Objects

When creating a generic object, make sure the gRPC framework doesn't already provide something similar.

Some common gRPC messages include:

• https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto
• https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto
• https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto

Provision of API Documentation

Always add comments to the Protobuf's objects. The Protobuf API is the contract/documentation between all microservices.

Comments, enforced by Buf's linter, must contain all information necessary to understand everything about the object.

Protobuf update

Protobuf binary compatibility != application compatibility

For example, changing a field name will not affect protobuf encoding or compatibility between applications that use proto definitions which differ only by field names. The binary protobuf encoding is based on tag numbers, so that is what you need to preserve, however, the specific language generated code will change and you will not use the same method to access the field.

Also, changing a field name will affect JSON representation if you use that feature.

Deprecated Fields

As a project evolves, its API changes. Over time, there are certain fields, types, methods or services that we don't want people to use anymore.

Instead of breaking the backward compatibility of the project's API, we will tag these elements with the [deprecated = true] option.

It tells other developers that the marked element should no longer be used and the element will be removed in the next major version change.

Where there is alternative functionality available for the functionality deprecated, add some extra comment to explain what would be a better alternative that serves the right behavior.

xxxxxxxxxx

 // At RPC level:

 rpc FooBar(FooBarRequest) returns (FooBarResponse) {

       option deprecated = true;

     };

 // At message level

 message Foo {

    option deprecated = true;

    string old_field = 1;

 }

 // At Field Level

 message Foo {

     string old_field = 1 [deprecated=true];

 }

Deprecated Fields Should Be Marked as Reserved

Fields can be removed, as long as the field number is not used again in your updated message type. Make the field number reserved, so that future users of your .proto can't accidentally reuse the number.

xxxxxxxxxx

 message Foo {

   reserved 2, 15, 9 to 11;

   reserved "foo", "bar";

   string field = 1;

 }

Semantic Versioning v. 1.0.0

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes.
• MINOR version when you add functionality in a backwards-compatible manner.
• PATCH version when you make backwards-compatible bug fixes.

When changing a MAJOR version, a logging change document must be created with all change details.

Breaking Changes

Protobuf APIs should be stable so as not to break consumers across repositories. To ensure backwards-compatibility, I recommend using Buf to check breaking changes every time a new Pull Request is opened.

When adding breaking changes, a new version structure should be created in the API following the Buf code style.

All microservices should be able to support multiple versions of a gRPC service.

API Owner

Responsibility for a service API lies with the team that maintains the associated microservice(s). API definitions are driven by the needs of the consumer(s) of the API.