An introduction to Protocol Buffers 3

I’ve spent most of my career building REST APIs. Then I had to build a gRPC service (gRPC is Google’s framework for calling functions on another server), and that meant learning Protocol Buffers. It looked daunting. It turns out Protobuf is easy to learn, full of useful features, and saves a lot of hand-written client code once the tooling is in place.

If you are looking into Protobuf for the first time, this post is the overview I wanted back then: the proto3 syntax, the building blocks, and the conventions worth following from day one.

What is Protobuf?

Protocol Buffers (Protobuf) is a free, open-source data format and toolchain for serializing structured data, which means turning it into bytes you can send over the network or save to disk, and reading it back. Google created it for internal use in 2001 and open-sourced it in 2008.

Google describes it like this:

Protobuf is a language-neutral, platform-neutral extensible mechanism for serializing structured data. It’s like JSON, except it’s smaller and faster, and it generates native language bindings. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

What problem does it solve?

Like JSON and XML, Protobuf is a way to exchange data between programs. You describe the data once in a .proto schema file, which is easy to read. Messages travel in a binary format, so they are smaller and faster to parse than text formats. It is a popular choice for RPC systems, where one service calls a function on another service over the network.

A few advantages of using Protobuf:

• Data is fully typed using a .proto schema.
• The protoc compiler generates the code that reads and writes your messages, in many languages.
• The schema can carry comments that document each field.
• Tools like buf can lint your schemas, detect breaking changes, and manage dependencies.

It also has costs worth knowing before you commit:

• Payloads are binary, so you can’t read them in logs or send them with plain curl. You need tools like grpcurl or buf curl.
• Browsers can’t call a gRPC service directly. You need gRPC-Web or Connect in between.
• Every client needs the .proto files, so you need a way to share them.
• Generated code has to be regenerated and kept in sync with the schema.

Syntax anatomy

The building block of Protobuf is the message: a named group of typed fields, much like a struct. Let’s break down a simple example, search.proto. You can find more details in the proto3 language guide.

syntax = "proto3"; // The version of the Protobuf syntax this file uses.

// Package name (think of it as a namespace for the schema).
package example.search.v1;

// Option annotation (more about this later).
option go_package = "github.com/victorpierredev/protoanatomy/gen/searchv1";

message SearchRequest { // message keyword + message name.
  string query = 1; // field type + field name + field number.
  int32 page_number = 2;
  int32 result_per_page = 3;
  repeated string parameters = 4; // the `repeated` keyword represents a list.

  enum SearchCategory {
    // When the field is not set, the zero value is the default.
    SEARCH_CATEGORY_UNSPECIFIED = 0; // The first enum value must be zero; later values need not be consecutive.
    SEARCH_CATEGORY_RENT = 1; // Prefix values with the enum name to avoid collisions.
    SEARCH_CATEGORY_BUY = 2;
    SEARCH_CATEGORY_LEASE = 3;
  }

  SearchCategory search_category = 5; // Enum as the type + field name + field number.
}

Option

Options are annotations that change how the code generator behaves. The ones you’ll see most are language-specific. If you are using Go, option go_package sets the full import path of the generated code, and the Go package name comes from the last part of that path. protoc-gen-go requires it (or an M flag on the command line). Java has option java_package, and other languages have their own.

Options can go at the top of the file, inside a message, or on a single field. The one you’ll reach for when a schema changes is deprecated. It tells readers and tools that new code should not use the field, and Java turns it into a @Deprecated annotation:

int32 page_number = 2 [deprecated = true];

Here is the complete list of available options. You can also create your own options, but most people never need to.

Scalar types

This is the field’s type, similar to types in most typed languages. See the list of supported scalar types.

Field name

Field names are written in snake_case. Another convention: if the field name contains a number, the number should appear after the letter instead of after an underscore. For example, use address1 instead of address_1.

Following these conventions pays off when generating code. Each code generator turns snake_case into its language’s own style, so page_number becomes PageNumber in Go.

For repeated fields, the field holds a list, so its name should be plural (e.g. repeated string middle_names).

Field numbers

This can be confusing at first, as it may look like a value assignment. Field numbers (often called tag numbers) identify each field in the binary format, so they must never be changed or reused once the message type is in use. They can range from 1 to 536,870,911 (2^29 - 1). A few things to keep in mind:

• Numbers from 1 to 15 use 1 byte when the message is encoded and should go to the most frequently used fields. Decide this when you first design the message, and never renumber an existing field to get a smaller number.
• From 16 to 2047, field numbers use 2 bytes.
• Numbers between 19000 and 19999 cannot be used, as they are reserved.

Enums

An enum lists the allowed values for a field. The first value must be zero, and it is what you get when the field is not set. Name it _UNSPECIFIED and never give it a business meaning, so “not set” can’t be mistaken for a real value. Enum values share the scope of the message or package around the enum, which is why each value is prefixed with the enum name.

Proto3 enums are also open: an older client can receive a number that was added after it was built. Always handle values you don’t recognize.

Default values

When a field isn’t set, reading it returns a default value for its type:

• bool defaults to false
• string defaults to ""
• bytes defaults to empty bytes
• numeric types (int32, int64, double and so on) default to 0
• repeated and map fields default to an empty list or map
• enums default to the first value, which must be zero
• message fields are simply not set, and how that looks depends on the language (nil in Go)

For a plain proto3 field, a default value isn’t even sent over the wire, so the receiver can’t tell a field set to 0 from one that was never set. That matters for updates, or whenever 0 is a real value.

Protobuf itself also has no built-in way to enforce value constraints. Say you have a field for a person’s age:

// age must be between 0 and 130
int32 age = 1;

Using messages inside other messages

In a real application, you will probably define several messages in the same .proto file (see defining a message type for the details). Each message becomes a type that other messages can use. For example, take an appointment booking app with this appointments.proto file:

syntax="proto3";

message Booking {
  string customer_name = 1;
  string service_type = 2;
  string notes = 3;

  // Note that `Date` is now an available type
  Date schedule = 4;
}

// Booking date that we want to use above.
message Date {
  int32 year = 1;
  int32 month = 2;
  int32 day = 3;
}

In real code, don’t write your own date type. Use google.protobuf.Timestamp for a moment in time, or google.type.Date from Google’s common types for a calendar date. I use a hand-rolled Date here only to keep the examples short.

Nested Types

It is also possible to define types within types (nesting), and you can do this with as many levels as needed. Expanding on the example above:

syntax="proto3";

message Booking {
  string customer_name = 1;
  string service_type = 2;
  string notes = 3;

  // Note that `Date` is now an available type
  Date schedule = 4;
  Status status = 5;

  // A nested message. Outside `Booking`, refer to it as `Booking.Status`.
  message Status {
    // Both enums are scoped to `Status`. Elsewhere, refer to them as
    // `Booking.Status.State` and `Booking.Status.PaymentStatus`.
    enum State {
      STATE_UNSPECIFIED = 0;
      STATE_PENDING = 1;
      STATE_CONFIRMED = 2;
      STATE_CANCELLED = 3;
    }

    enum PaymentStatus {
      PAYMENT_STATUS_UNSPECIFIED = 0;
      PAYMENT_STATUS_PENDING = 1;
      PAYMENT_STATUS_PAID = 2;
      PAYMENT_STATUS_REJECTED = 3;
    }

    // The field numbers are also relative to the `Status` message.
    State state = 1;
    PaymentStatus payment_status = 2;
  }
}

// Booking date that we want to use above.
message Date {
  int32 year = 1;
  int32 month = 2;
  int32 day = 3;
}

I use one State enum instead of two booleans like confirmed and cancelled. Two booleans allow a booking that is both confirmed and cancelled, and an enum leaves room for new states later, such as rescheduled.

Importing types

Another way to use message types defined in other files is to import them. In this example I also declare a package ¹. Going back to the booking example, imagine these two files in different directories:

.
├── api
│   └── v2
│       └── appointments.proto
└── types
    └── v1
        └── date.proto

• date.proto

syntax = "proto3";

// Packages prevent name clashes between types, much like namespaces in
// other languages. They are optional, but always declare one and include
// a version. In a real project, add a unique prefix too (`acme.types.v1`).
// In Go, the proto package is ignored: go_package decides the Go package.
package types.v1;

message Date {
  int32 year = 1;
  int32 month = 2;
  int32 day = 3;
}

• appointments.proto

syntax = "proto3";

package api.v2;

// The import path is the file's path relative to the --proto_path (-I)
// root, not the package name.
import "types/v1/date.proto";

message Booking {
  string customer_name = 1;
  string service_type = 2;
  string notes = 3;

  // Because this file imports date.proto, the type is available.
  // Because date.proto declares a package, the type name must include it.
  // Without a package, it would be `Date schedule = 4;`
  types.v1.Date schedule = 4;
}

Defining a service

Since I came to Protobuf through gRPC, one more piece: a service block declares the functions a server offers, with messages as their input and output. gRPC’s code generator turns it into a client and a server interface.

service SearchService {
  rpc Search(SearchRequest) returns (SearchResponse);
}

Changing a schema safely

Field numbers matter so much because a schema keeps changing after clients start using it. Old clients keep reading new messages, and new clients keep reading old ones. A few rules keep that working:

• Adding a field with a new number is always safe. Old clients ignore fields they don’t know about.
• Never change a field’s number or type.
• When you delete a field, reserve its number and name so nobody reuses them by accident. A reused number can make old clients misread the new field’s data as the old field, or fail to parse it. The same goes for enum values.
• Renaming a field doesn’t change the binary format, but it does change the JSON form of the message, so treat it as breaking if anything uses JSON.

Here is search.proto a few releases later:

message SearchRequest {
  reserved 3; // Was result_per_page, now deleted. The number can never be reused.
  reserved "result_per_page"; // Reserve the name too.

  string query = 1;
  int32 page_number = 2 [deprecated = true]; // Step one of removing a field.
  repeated string parameters = 4;
  SearchCategory search_category = 5; // The enum is unchanged.
  int32 page_size = 6; // A new field gets a new number.
}

buf breaking can check these rules for you in CI by comparing your schema against your main branch (buf breaking --against '.git#branch=main'). If you really need a breaking change, publish it in a new package version, such as api.v3, next to the old one, and move clients over.

Wrapping up

Coming from REST, I expected a steep learning curve and came out with a handful of rules. Messages hold typed fields. Every field gets a number that never changes and is never reused. Enums start at an unspecified zero. Types can be nested or imported as the schema grows, and reserved keeps old clients safe when it changes. If you build services that talk to each other, convert one API contract to a .proto file and see how it fits.