The Buffrs Book

Buffrs Logo

Buffrs is a protocol buffers package manager. Buffrs downloads your projects protocol buffer dependencies, creates distributable packages for libraries and apis, and uploads them to a package registry.

Sections

Getting Started

To get started with Buffrs, install it and set up your first package.

Buffrs Guide

An in-depth guide on how to use Buffrs to distribute protocol buffers.

Buffrs Integrations

Learn how to integrate Buffrs into existing build tools for a seamless development experience.

Buffrs Reference

Detailed aspects of Buffrs behavior, including advanced configuration and troubleshooting resources.

Buffrs Registry Reference

Advanced resources on the Buffrs Registry.

Buffrs Commands

Detailed documentation of the commands of the Buffrs CLI.

Getting Started

To get started with buffrs, install it and set up your first package.

Installation

Install Buffrs

The easiest way to get buffrs is to install the current stable release from crates.io using:

cargo install buffrs

As of right now you are required to authenticate yourself against your private artifactory instance (which will be replaced by the Buffrs Registry in Q4 2023).

Logging in to your instance is done using the following buffrs command:

buffrs login --registry https://<organization>.jfrog.io/artifactory

You will be prompted for an artifactory identity token which you can create within the artifactory user interface or programmatically through terraform.

Build and Install Buffrs from Source

As alternative installation method you can clone the Buffrs Repository and install it locally using Cargo (cargo install --path .).

First Steps with Buffrs

This section gives you a quick intro to the buffrs command line interface. Let us take a look at its ability to declare protocol buffer dependencies and to publish new packages to the registry.

To initialize a new project with Buffrs, use buffrs init:

$ mkdir web-server && cd web-server
$ buffrs init --api

Note: By omitting the --api flag (or --lib flag respectively) you instruct Buffrs to not declare a local package and setup the project to be a consumer-only (e.g. a server implementation).

$ tree .
.
├── Proto.toml
└── proto
    └── vendor

2 directories, 1 file

This is all we need to get started. Now let’s check out the newly created Proto.toml:

[package]
name = "web-server"
version = "0.1.0"
type = "api"

[dependencies]

This is called a Buffrs Manifest, and it contains all of the metadata that Buffrs needs to know about your package to install dependencies and distribute your protocol buffers as a package.

Let us define a dependency of the webserver on a hypothetical library called user in the datatypes repository.

This is done by invoking buffrs add:

$ buffrs add --registry https://your.registry.com datatypes/user@=0.1.0

The result is a dependency in the Proto.toml:

[dependencies.user]
version = "=0.1.0"
repository = "datatypes"
registry = "https://your.registry.com/"

Going further

For more details on using Buffrs, check out the Buffrs Guide

Buffrs Guide

This guide will give you all that you need to know about how to use Buffrs to develop and consume protocol buffer packages.

Why Buffrs Exists

Modern gRPC based software platforms make extensive use of protocol buffers to define and implement inter-service APIs. While the open-source protocol buffers ecosystem makes it very easy to get started with a few services and a handful of messages, recurring difficulties occur with scaling gRPC APIs beyond immediate repository boundaries for two main reasons:

Limited tooling for packaging, publishing, and distributing protocol buffer definitions.
The lack of widely-adopted best practices for API decomposition and reuse.

To overcome these problems we developed Buffrs - a package manager for protocol buffers. Using Buffrs, engineers package protocol buffer definitions, publish them to a shared registry, and integrate them seamlessly into their projects using versioned API dependencies and batteries-included build tool integration.

For a detailed differentiation against existing approaches (like buf, bazel and git submodules) and architecture deep dive take a look at the announcement post.

The Framework

To understand the distribution and decomposition framework that Buffrs provides it is useful to understand which properties of API management systems are desirable and why. Key aspects include:

Versioning

A versioning scheme for APIs — similar to versioned library dependencies — explicitly encodes compatibility properties. This allows developers to make either backwards-compatible or breaking changes and encode the compatibility guarantees in the API version: a minor version upgrade “just works”, while a new major version API may require manual migration/adaption in the consuming server/client implementations.

Source Compatibility

Given versioned protocol buffer APIs with explicit compatibility guarantees, it is desirable to have a system in which wire-format compatible APIs are also source-code compatible. This means that engineers can update minor patches automatically and that APIs that build upon the same protocol buffer types can be used with the same generated code types. This is especially important in strict languages like Rust (due to, e.g., the orphan rule).

Composition

Enabling engineers to reuse and combine code in order to build new systems from existing building blocks is a key feature. A common composition scheme with protocol buffers is to use a set of base messages and data types across many APIs.

Discoverability

Before engineers can reuse and compose protocol buffers, they need to be able to discover and understand what APIs exist and how to use them. Discoverability is a significant accelerator for engineering productivity and helps developers stay abreast of the evolution of APIs and architecture.

Protocol Buffers as a First Class Citizen

Buffrs decided to take a unique approach compared to other protocol buffer management systems (or systems capable of distributing protocol buffers) like buf and bazel. Buffrs is essentially a package manager for protocol buffers. Protocol buffers are treated as a first class citizen within Buffrs – which means that they are seen as distributable units called packages.

Complex software projects frequently turn out to depend on different versions of the same APIs over time and individual components in those systems may have diverging compatibility guarantees. Internal services may break their API backwards compatibility, way more frequent than external gateways that serve millions of users.

The three fundamental ideas of Buffrs to enable a stable management approach for scenarios like the above are:

Buffrs Packages

A closed and meaningful unit of protocol buffers which enables either productivity through shared types (e.g., Google's google.protobuf.Timestamp) or describes a domain / API (e.g., service GeoLocator).

Buffrs Registry

A central registry for managing and hosting packages, documentation, enabling engineers to search and find and to implement access control.

Local Code Generation

The last block is local code generation. This enables projects to freely diverge in their actual implementations by choosing a code generation implementation which fits their specific needs while still maintaining complete wire compatibility. This prevents language lock-ins and makes any version of any package usable for any language that exists today or might exist in the future (given it has a protobuf code generator).

Package Types

Buffrs makes distinctions between two different packages:

[package]
type = "lib" | "api"

This is used in order to fuel composition and type reuse across APIs and thus enable shared types + wire compatibility.

`lib` – Libraries

Libraries contain atomic and composite type definitions that describe a domain. (e.g. physics, auth, time etc.). This pattern is really useful for scaling systems and maintaining dozens of APIs as they can share types and thus get the aforementioned benefits of source code and wire compatibility for free.

An example of a proto library named time that depends on google:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", ... }

syntax = "proto3";

package time;

import "google/timestamp.proto";

/// A timestamp wrapper for various formats
message Time {
  oneof format {
    string rfc3339 = 1;
    uint64 unix = 2;
    google.protobuf.Timestamp google = 3;
    ..
  }
}

`api` – APIs

APIs are the next logical building block for real world systems – they define services and RPCs that your server can implement. You can use the aforementioned libraries to fuel your development / API definition experience.

A good example of an API could be an imaginary logging service that makes use of the just declared time.Time:

[package]
name = "logging"
type = "api"
version = "1.0.0"

[dependencies]
time = { version = "=1.0.0", ... }

syntax = "proto3";

package logging;

import "time/time.proto";

service Logging {
  rpc critical(LogInput) returns (LogOutput);
  rpc telemetry(LogInput) returns (LogOutput);
  rpc healthiness(HealthInput) returns (HealthOutput);
}

message LogInput { string context = 1; time.Time timestamp = 2; }
message LogOutput { }

message HealthInput { bool db = 1; }
message HealthOutput { }

Creating a Package

Creating a new, consumable, Buffrs package can be done in four steps:

Initialize a project using buffrs init
Set metadata, package type, version correctly and declare the dependencies that you want to use.
Declare message, enum and service types you want to publish in the newly created proto folder using .proto files.
Publish to a registry using buffrs publish.

Example: Publishing `physics`

Initialize Your Project

Start by initializing a new Buffrs project using the buffrs init command. This will set up the basic structure for your package and allow you to manage your package's metadata and dependencies.

mkdir physics
cd physics
buffrs init --lib

Define Package Metadata and Dependencies

In your project folder, locate the Proto.toml file. This is where you'll specify your package's metadata and dependencies. Open this file in a text editor.

Here's an example Proto.toml file:

# Package metadata
[package]
package = "physics"
version = "1.0.0"
type = "lib"
description = "A library containing physic related types"

# Declare dependencies (none in this case)
[dependencies]

Define Message, Enum, and Service Types

Inside your project directory, buffrs init created a proto folder. This is where you will store your .proto files that define your library or api.

An example file structure:

physics
├── Proto.toml
└── proto
    ├── temperature.proto
    ├── mass.proto
    └── vendor

Write your Protocol Buffer definitions in these .proto files. Here's a simple example of the temperature.proto file that could be in a physics library:

syntax = "proto3";

package physics.temperature;

// Define temperature units
enum TemperatureUnit {
  CELSIUS = 0;
  FAHRENHEIT = 1;
  KELVIN = 2;
}

// Define a message for temperature
message Temperature {
  double value = 1;            // Temperature value
  TemperatureUnit unit = 2;    // Temperature unit (Celsius, Fahrenheit, Kelvin)
}

Publish Your Package

Once you've set up your Buffrs package and defined your Protocol Buffer definitions, you can publish it to a registry using the buffrs publish command. Make sure you're logged in to the registry if required.

buffrs publish --registry https://your.registry.com --repository tutorial

Your package will be uploaded to the registry, and others can now consume it using Buffrs.

Congratulations! You've successfully published your Buffrs package. Other developers can now use your Protocol Buffer definitions by adding your package as a dependency in their Buffrs projects.

That's it! You've created a Buffrs package that others can easily consume. Remember to keep your package up-to-date and well-documented to make it even more valuable to the community.

Consuming Packages

As described in the package types section you can declare dependencies through your project's Proto.toml. This is true for both libraries and APIs. But what if you want to implement your server and you need to consume buffrs packages that contain your type and service definitions?

So there are three scenarios in which you would want to depend on other packages:

a) You are defining a library and you want to make use of an external type coming from another library (e.g. google for basic types such as google.None). b) You are defining an API and you want to use libraries to reuse common types for that domain (e.g. time or physics) c) You are implementing your server and you want to get access to API definitions to generate bindings.

The good news are: They are all achieved in a similar fashion. You make use of the [dependencies] key in your manifest to declare the packages that your projects needs – either to publish another package or to compile to protos to bindings.

Examples

Libraries & APIs

This section is identical for libraries and APIs.

An example of the time library reusing the google library:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository> }

Running buffrs install yields you with the following filesystem:

time
├── Proto.toml
└── proto
    ├── time.proto
    └── vendor
        ├── time
        ├   └── ..
        └── google
            ├── any.proto
            ├── ..
            ├── struct.proto
            └── timestamp.proto

You can now develop your library and publish it using buffrs publish.

Servers

If you want to implement your server and thus use e.g. a logging API the only major difference is the lack of the [package] section in your manifest.

[dependencies]
logging = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository> }

Running a buffrs install yields you the very same as above, except for the omitted local package and the logging dependency instead of time.

.
├── Proto.toml
└── proto
    └── vendor
        └── logging
            └── logger.proto

Local Dependencies

When working on larger projects or in monorepo projects setups you may find yourself in the situation to consume a locally defined buffrs project.

Imagine the following project setup:

mono
├── build.rs
├── Cargo.toml
├── Proto.toml
├── Proto.toml
├── proto
|   └── mono.proto
└── src
    └── main.rs

In this scenario the buffrs project for mono-api and the cargo project for the mono-server are setup in the very same directory, which is totally fine as long as this server does not require other buffrs api packages to be compiled!

Problem

Adding a dependency on other (unrelated api packages to mono-api) is complicated in the above scenario because its not clear to buffrs whether you are trying to reuse third party api definitions for your api or just wanting to install protos for compilation.

Hence buffrs will throw you an error if you try to publish an api package with dependencies on other apis!

Error:   × failed to publish `mono-api` to `http://<registry-uri>/<repository>`
  ╰─▶ depending on API packages is not allowed

Solution

Gladly buffrs offers a builtin solution for this! You can separate the mono-api buffrs package (used to publish your api) from the mono-server buffrs projects (used to install protos for compiling the server).

A monorepo setup here could look like this:

mono
├── mono-api
|   ├── Proto.toml
|   └── proto
|       └── mono.proto
└── mono-server
    ├── build.rs
    ├── Cargo.toml
    ├── Proto.toml
    ├── proto
    |   └── vendor
    └── src
        └── main.rs

Where mono/mono-api/Proto.toml has this content:

edition = "0.9"

[package]
type = "api"
name = "mono-api"
version = "0.1.0"

And mono/mono-server/Proto.toml has this content:

edition = "0.9"

[dependencies]
mono-api = { path = "../api" }
third-party-api = { version = "=1.0.0", repository = "some-repo", registry = "http://..." }

This enables you to:

Independently publish mono-api using buffrs publish / buffrs package
Independently declare dependencies for mono-server

Publishing with Local Dependencies

Since version 0.12, buffrs automatically handles publishing packages with local dependencies:

cd mono-server
buffrs publish --registry http://... --repository my-repo

Buffrs will:

Recursively resolve all local dependencies (and their transitive dependencies)
Publish each local dependency in topological order
Rewrite local references in the published manifest to point to the registry

For example, if mono-server depends on mono-api (local), buffrs first publishes mono-api to the registry, then publishes mono-server with the dependency updated to reference the registry location instead of the local path.

This makes it seamless to develop locally while maintaining publishable packages.

Workspaces

Since version 0.12, buffrs supports workspaces for managing multiple related packages in a single repository.

What is a Workspace?

A workspace is a collection of buffrs packages that share a common root directory. This is useful when maintaining multiple API versions, splitting functionality across packages, or managing shared libraries alongside consuming packages.

Creating a Workspace

To create a workspace, add a [workspace] section to your root Proto.toml:

edition = "0.12"

[workspace]
members = [
  "packages/common",
  "packages/api-one",
  "packages/api-two",
]

Each member is a separate buffrs package with its own Proto.toml manifest. The workspace root Proto.toml should not contain a [package] section - it only defines the workspace structure.

Directory Structure

A typical workspace structure:

my-workspace/
├── Proto.toml              # Workspace root manifest
└── packages/
    ├── common/
    │   ├── Proto.toml      # Package manifest
    │   └── proto/
    │       └── common.proto
    ├── api-one/
    │   ├── Proto.toml
    │   └── proto/
    │       └── api.proto
    └── api-two/
        ├── Proto.toml
        └── proto/
            └── api.proto

Workspace Commands

Installing Dependencies

From the workspace root:

buffrs install

This installs dependencies for all workspace members, handling inter-workspace dependencies automatically.

Publishing

From the workspace root:

buffrs publish --registry http://... --repository my-repo

Buffrs automatically:

Resolves dependencies across all workspace members
Publishes packages in topological order (dependencies first)
Handles local dependencies within the workspace
Ensures each package is published only once, even if multiple members depend on it

Uninstalling

From the workspace root:

buffrs uninstall

Clears vendor directories for all workspace members.

Inter-workspace Dependencies

Workspace members can depend on each other using local path references:

# packages/api-two/Proto.toml
edition = "0.12"

[package]
type = "api"
name = "my-api-two"
version = "2.0.0"

[dependencies]
"my-common" = { path = "../common" }

When publishing the workspace, buffrs automatically publishes my-common first, then publishes my-api-two with the dependency reference updated to point to the registry.

Package-only Commands

Some commands are designed for package-level operations and cannot run from a workspace root:

buffrs add
buffrs remove
buffrs package
buffrs lint
buffrs list

These commands will provide a clear error message directing you to run them from a package directory within the workspace.

Benefits

Workspaces provide:

Single-command publishing: No need to manually navigate directories or worry about dependency order
Automatic deduplication: Shared dependencies are published once
Development efficiency: Develop and test changes across multiple packages simultaneously
Monorepo support: Natural fit for monorepo setups with multiple API packages

Import System

To reuse already defined messages, protobuf files can be imported from both dependencies and the managed package itself. Unique identification of the files is made available through the name of the package as declared in Proto.toml which is used as root of the imports.

For a dependency units:

units
├── Proto.toml
├── weight.proto
└── length.proto

and a package named physic:

physic
├── Proto.toml
└── proto
    ├── root.proto
    ├── length.proto
    └── calculations
        ├── distance.proto
        └── graph.proto

messages can be imported from both packages relative to their root:

// root.proto
syntax = "proto3";

package physic;

import "physic/length.proto";
import "physic/calculations/distance.proto";
import "units/length.proto";

message Lengths {
    units.Meter meter = 1;
    physic.Parsec parsec = 2;
    physic.calculations.Haversine = 3;
}

Project Layout

To get an understanding of the project layout that buffers uses it is helpful to start in a clean manner and introspect the outcome.

Lets create a new clean directory initialize for our physic library.

$ mkdir physic
$ cd physic
$ buffrs init --lib

This will initialize the following project structure:

physic
├── Proto.toml
└── proto
    └── vendor

This will create the Proto.toml file which is the manifest file that buffrs uses. The proto directory, which is the source directory for all your protocol buffer definitions and the proto/vendor directory, which contains external protocol buffers.

Important: The vendor directory is managed by Buffrs, all manual changes will be overridden / can cause unreproducible behavior.

Manifest (`Proto.toml`) vs Lockfile (`Proto.lock`)

Manifest – `Proto.toml`

Purpose: The Manifest (Proto.toml) serves as a specification file for the current package. It includes metadata, dependencies, and other package-related information.

Contents:

Package Name: A unique name for the package.
Version: Specifies the package version.
Type: Specifies the type of this package (see Package Types).
Description: Provides a brief description of the package.
Metadata: Additional metadata like package category, tags, and any other relevant package details.
Dependencies: Contains package names, version constraints, registry locations etc of the dependencies of the current package.

Usage: The Proto.toml is used to define the package's characteristics, metadata, and its dependencies, making it a comprehensive specification format. This file is included in compressed package artefacts and distributed alongside. For a usage guide see Creating A New Package.

Lockfile – `Proto.lock`

Purpose: The Lockfile (Proto.lock) is a separate, autogenerated and automanaged, file that records the exact versions of packages, including their transitive dependencies, which have been successfully resolved and installed.

Contents: It contains a detailed record of the package versions used, the registry or package source used during the installation and cryptographic hashes to ensure package integrity.

Usage: The lockfile is crucial for ensuring the reproducibility of installations. It guarantees that the same package versions are installed, regardless of changes in the upstream package registry, ensuring consistency across different installations.

Buffrs Home

$HOME/.buffrs

The buffrs home directory is a place for global configuration and data belonging to buffrs such as credentials and a cache. You should not need to interact with this folder manually.

The home directory that is used by buffrs can be configured via the BUFFRS_HOME environment variable. This enables you do override the default location in case you want to keep your home directory clean.

Integrations

Having a standalone distribution system and package manager for protocol buffers is the first step towards solving the problem that engineers are facing.

Ensuring a works out of the box experience through integrating into common build systems is the next one.

Currently we are providing the following integrations:

If you are missing your favorite build system, please create an issue on GitHub to request it – or ideally help us by building or researching parts the integration!

Migration Guide

Once you read the initial guide on how buffrs works and how to use it, you can follow this guide to migrate your existing protobuf code into buffrs and use it as the distribution mechanism!

Continuous Integration

Continuous Integration

To utilize continuous integration for your Buffrs package (e.g. to automate code review and publishing of your packages) you can utilize the following templates for GitHub Actions and GitLab CI:

GitHub Actions

name: Buffrs

on:
  push:
    branches:
      - '*'
  tags:
    - '*'

env:
  REGISTRY: https://<org>.jfrog.io/artifactoy
  REPOSITORY: your-artifactory-repo

jobs:
  verify:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly

      - name: Set up Rust environment
        run: |
          rustup target add aarch64-unknown-linux-gnu
        shell: bash

      - name: Verify
        run: |
          cargo install --force buffrs
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs lint
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

  publish:
    runs-on: ubuntu-latest
    needs: build
    if: startsWith(github.ref, 'refs/tags/')

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Publish on tag
        run: |
          cargo install --force buffrs
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs publish --registry $REGISTRY --repository $REPOSITORY
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

GitLab CI

stages:
  - verify
  - publish

variables:
  TOKEN: $BUFFRS_TOKEN  # Your secret artifactory token
  REGISTRY: https://<org>.jfrog.io/artifactory
  REPOSITORY: your-artifactory-repo

verify:
  stage: verify
  script:
    - cargo install buffrs
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs lint
  only:
    - branches

publish:
  stage: publish
  script:
    - cargo install buffrs
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs publish --registry $REGISTRY --repository $REPOSITORY
  only:
    - tags

Buffrs Reference

This section covers detailed documentation on the Buffrs Ecosystem / Framework.

Index

Editions

Editions of buffrs mark a specific evolutionary state of the package manager. The edition system exists to allow for fast development of buffrs while allowing you to already migrate existing protobufs to buffrs even though it has not yet reached a stable version.

Editions can be either explicitly stated in the Proto.toml or are automatically inlined once a package is created using buffrs. This ensures that you dont need to care about them as a user but get the benefits.

Note: If you release a package with an edition that is incompatible with another one (e.g. if 0.7 is incompatible with 0.8) you will need to re-release the package for the new edition (by bumping the version, or overriding the existing package) to regain compatibility.

You may see errors like this if you try to consume (or work on) a package of another edition.

Error:   × could not deserialize Proto.toml
  ╰─▶ TOML parse error at line 1, column 1
        |
      1 | edition = "0.7"
        | ^^^^^^^^^^^^^^^
      unsupported manifest edition, supported editions of 0.8.0 are: 0.8

Canary Editions

edition = "0.7"

Canary editions are short-lived editions that are attached to a specific minor release of buffrs in the 0.x.x version range. The edition name contains the minor version it is usable for. E.g. the edition 0.7 is usable / supported by all 0.7.x buffrs releases. Compatibility beyond minor releases is not guaranteed as fundamental breaking changes may be introduced between editions.

Specifying Dependencies

Dependencies are declared in the [dependencies] section of the Proto.toml manifest. Each entry maps a dependency package name to a dependency specification object.

Remote Dependencies

Remote dependencies are downloaded from an Artifactory registry during buffrs install.

[dependencies]
my-package = { registry = "https://your.registry/artifactory", repository = "my-repo", version = "1.2.3" }

The version field must be an exact semantic version (e.g. "1.2.3"). Version ranges or operators (^, ~, <, >) are not currently supported.

Use buffrs add to add a remote dependency from the command line:

buffrs add --registry https://your.registry/artifactory my-repo/my-package@1.2.3

Local Dependencies

Local dependencies are resolved from the local filesystem relative to the manifest. They are useful for multi-package repositories where packages depend on each other without going through a remote registry.

[dependencies]
my-lib = { path = "../my-lib" }

The path field is a relative path from the manifest file to the dependency's root directory (the directory containing the dependency's Proto.toml).

See Local Dependencies for more information.

Lockfile

After adding or modifying dependencies in the manifest, run buffrs install to resolve and lock them. The lockfile (Proto.lock) records the exact resolved versions and checksums and should be committed to version control.

See Manifest vs Lockfile for more information.

The Manifest Format

The Buffrs manifest file is named Proto.toml and is placed at the root of a Buffrs package or workspace. It is written in TOML format.

Fields

`edition`

edition = "0.12"

The edition field declares which edition of the Buffrs manifest format is being used. The edition is tied to the minor version of buffrs that introduced it, so "0.12" means the edition introduced with buffrs 0.12.x.

Editions are set automatically when you create a new package with buffrs init or buffrs new and are validated on load. If the edition is incompatible with the installed buffrs version, an error is reported.

See Editions for more information.

`[package]`

The [package] section declares metadata about the current package. It is optional when using a workspace-only manifest.

[package]
type = "api"
name = "my-api"
version = "1.0.0"
description = "My API package"   # optional

Field	Required	Description
`type`	yes	Package type: `"api"`, `"lib"`, or `"impl"`
`name`	yes	Package name – must be lowercase ASCII and dashes (see Package Name Specifications)
`version`	yes	Semantic version of the package (e.g. `"1.2.3"`)
`description`	no	A short human-readable description of the package

See Package Types for more information.

`[dependencies]`

The [dependencies] section lists the packages that this package depends on. Each entry maps a package name to a dependency specification.

Remote dependency

[dependencies]
my-package = { registry = "https://your.registry/artifactory", repository = "my-repo", version = "1.2.3" }

Field	Required	Description
`registry`	yes	URL of the Artifactory registry
`repository`	yes	Name of the repository within the registry
`version`	yes	Exact version to install (e.g. `"1.2.3"`)

Local dependency

[dependencies]
my-lib = { path = "../my-lib" }

Field	Required	Description
`path`	yes	Relative path to the local package directory

See Local Dependencies and Specifying Dependencies for more details.

`[workspace]`

The [workspace] section turns the manifest into a workspace root. It is mutually exclusive with the [package] section (a file can be either a workspace root or a package, not both).

[workspace]
members = ["pkg1", "pkg2"]

Field	Required	Description
`members`	yes	List of relative paths to workspace member package directories

See Workspaces for more information.

Examples

Minimal implementation manifest

edition = "0.12"

API package with a remote dependency

edition = "0.12"

[package]
type = "api"
name = "my-api"
version = "0.1.0"

[dependencies]
common-types = { registry = "https://your.registry/artifactory", repository = "protos", version = "1.0.0" }

Library package

edition = "0.12"

[package]
type = "lib"
name = "common-types"
version = "1.0.0"
description = "Shared protobuf type definitions"

Workspace manifest

edition = "0.12"

[workspace]
members = ["api", "lib"]

The Lockfile Format

The Buffrs lockfile is named Proto.lock and is placed at the root of a Buffrs package or workspace. It is written in TOML format and is managed automatically by buffrs install.

The lockfile should be committed to version control. It ensures that all contributors and CI environments install the exact same dependency versions, including transitive dependencies.

Structure

version = 3

[[package]]
name = "my-dep"
version = "1.2.3"
registry = "https://your.registry/artifactory"
repository = "my-repo"
digest = "sha256:abc123..."

[[package]]
name = "transitive-dep"
version = "0.4.1"
registry = "https://your.registry/artifactory"
repository = "my-repo"
digest = "sha256:def456..."

`version`

The lockfile format version. This is managed automatically by buffrs.

`[[package]]`

Each [[package]] entry records one resolved dependency (direct or transitive). Fields:

Field	Description
`name`	Package name
`version`	Exact resolved version
`registry`	Registry URL the package was downloaded from
`repository`	Repository within the registry
`digest`	SHA-256 checksum of the downloaded package archive (prefixed with `sha256:`)

Lockfile Interaction

The lockfile is automatically created or updated when running buffrs install. You should not need to edit it manually.

To obtain the list of locked files as JSON (useful for scripted or sandboxed installations), use buffrs lock print-files.

See Manifest vs Lockfile for more information on the relationship between the manifest and the lockfile.

Configuration

Authentication

Buffrs uses a local credential storage for authenticating with registries. The login command can be used to add new credentials to the storage. Once saved, credentials are automatically used for authenticating with the registry they are associated with. Registries are identified by their URL.

Note that credentials are optional, if they are missing for a given registry URL, no authentication is attempted.

TLS configuration

Buffrs will automatically pick up the SSL_CERT_FILE environment variable if it's been set, and attempt to use the native subsystem to parse and load the specified root certificate into the certificate store. No additional configuration is needed to apply custom root certificates.

Proxy support

Buffrs will automatically pick up on HTTP_PROXY and HTTPS_PROXY environment variables if they've been set, and use the specified proxy URLs for the associated remote requests. No additional configuration is needed.

Environment Variables

Buffrs reads the following environment variables at runtime.

`BUFFRS_HOME`

The home directory for Buffrs data, including the credentials store. Defaults to ~/.buffrs if not set.

See Buffrs Home for more information.

`BUFFRS_CACHE`

Path to the package cache directory. When set, Buffrs uses this directory to cache downloaded packages so that subsequent installs do not require network access. This is particularly useful in sandboxed build environments such as Nix.

`BUFFRS_VERBOSE`

Set to true to enable verbose (debug-level) logging output. Equivalent to passing the -v / --verbose flag on the command line.

`SSL_CERT_FILE`

Path to a custom root certificate file. When set, Buffrs loads the specified certificate into the TLS certificate store. This is useful in corporate environments that use a custom CA.

`HTTP_PROXY` / `HTTPS_PROXY`

Proxy URLs for HTTP and HTTPS requests respectively. When set, Buffrs routes outgoing requests through the specified proxy server.

Package Name Specifications

Buffrs package names must conform to the following rules:

Must be at least 1 character long.
Must be at most 128 characters long.
Must start with an ASCII lowercase alphabetic character (a–z).
Must consist only of ASCII lowercase letters (a–z) and hyphens (-).

In other words, valid package names match the regular expression ^[a-z][a-z-]{0,127}$.

Examples

Valid names:

physics
common-types
my-api

Invalid names:

Physics – uppercase letters not allowed
my_api – underscores not allowed
1my-api – must start with a letter
(empty string) – must be at least one character

Relationship to Protocol Buffer Package Names

The Buffrs package name is used as the required prefix for all Protocol Buffer package declarations inside the package. For example, a Buffrs package named physics must declare protocol buffer packages matching physics or physics.*.

See Protocol Buffer Rules for more information.

Protocol Buffer Rules

This specification defines rules enforced by Buffrs to prevent package colisions, and provide uniformity and transparency for package consumers.

Rules with the 00XX code define the package and filesystem layout, where as rules with a 01XX code enforce certain protocol buffer definition rules.

`0000` – Package Name Prefix / Symmetry

Enforces that the Buffrs Package ID is used as the prefix for all protocol buffer package declarations.

So given a Buffrs Package with the ID physics this enforces that the package only contains protocol buffer package declarations matching physics|physics.*;

A violation would cause type colisions and ambiguity when trying to resolve a type.

`0010` – Sub-Package Declaration

Enforces that subpackages are declared through a sensible folder structure. Given a Buffrs Package with the ID physics the protocol buffer file that declares package physics.units; has to be called proto/units.proto.

Nested subpackages are represented / grouped through folders. So if one wants to declare package physics.units.temperature; the respective file must be located at proto/units/temperature.proto.

`0020` – Root Package Declaration

Enforces that only one file at a time declares the root package.

Namely: If a Buffrs Package with the ID physics is defined, the proto/physics.proto must declare the the same package in the protocol buffer syntax through package physics;.

Buffrs Commands

This section covers detailed documentation on operating the Buffrs CLI. Some of the content may also apply to the library, particularly the command module.

The goal is to provide sufficient information to use the CLI effectively to manage a Buffrs project, and also capture context and behaviour that may otherwise be only expressed in code.

The help command should also provide useful information and is expected to generally be more up-to-date.

Index

General Commands

General commands interface with the CLI itself, in order to obtain access to built-in help or version information.

Index

buffrs

The official Buffrs command-line interface.

Synopsis

buffrs

Description

When invoked without any arguments, the Buffrs binary defaults to printing out help information to the standard output.

This is equivalent to buffrs help, or invoking with the -h or --help flags.

Providing the -V or --version flags prints the current version of buffrs.

Output

Modern protobuf package management

Usage: buffrs [OPTIONS] <COMMAND>

Commands:
  init       Initializes a buffrs setup
  new        Creates a new buffrs package in the current directory
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  list       Lists all protobuf files managed by Buffrs to stdout
  login      Logs you in for a registry
  logout     Logs you out from a registry
  lock       Lockfile related commands
  help       Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose  Enable verbose logging [env: BUFFRS_VERBOSE=]
  -h, --help     Print help
  -V, --version  Print version

buffrs help

Prints out information about how to use the CLI.

Synopsis

buffrs help [command]

Description

When called by itself, this command lists all the supported commands along with a brief description.

When called with a command argument, it will provide specific help for that command.

Passing the -h or --help flags is equivalent to invoking this command.

Examples

> buffrs help
Modern protobuf package management

Usage: buffrs [OPTIONS] <COMMAND>

Commands:
  init       Initializes a buffrs setup
  new        Creates a new buffrs package in the current directory
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  list       Lists all protobuf files managed by Buffrs to stdout
  login      Logs you in for a registry
  logout     Logs you out from a registry
  lock       Lockfile related commands
  help       Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose  Enable verbose logging [env: BUFFRS_VERBOSE=]
  -h, --help     Print help
  -V, --version  Print version

> buffrs help init
Initializes a buffrs setup

Usage: buffrs init [OPTIONS] [PACKAGE]

Arguments:
  [PACKAGE]  The package name used for initialization

Options:
      --lib      Sets up the package as lib
      --api      Sets up the package as api
  -h, --help     Print help
  -V, --version  Print version

Manifest Commands

Manifest commands manage the package's manifest file, which declares important metadata like the package name, version and dependencies.

These commands can be used to include, remove and update dependencies while keeping the lockfile in sync.

Index

buffrs add

Adds a new dependency to the package manifest.

Synopsis

buffrs add --registry <REGISTRY> <DEPENDENCY>

Description

The add command is the recommended way to include a new dependency in the current package. It modifies the local manifest file and will overwrite a pre-existing entry for the same dependency package if it exists.

Dependency locator format

The dependency should be specified with the repository, name and version according to the following format:

<repository>/<package>[@<version>]

Note: the version can be omitted (or set to @latest), in which case it will default to the latest version of this artifact in the registry.

The repository name should adhere to lower-kebab case (e.g. my-buffrs-repo). The package name has its own set of constraints as detailed in Package Name Specification. When specified, the version must adhere to the Semantic Version convention (e.g. 1.2.3) -- see SemVer compatibility for more information.

Currently there is no support for resolving version operators but the specific version has to be provided. This means ^1.0.0, <2.3.0, ~2.0.0, etc. can't be installed, but =1.2.3 has to be provided.

Lockfile interaction

Currently adding a new dependency won't automatically update the lockfile (Proto.lock). This is planned to change, but for now follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

buffrs remove

Removes an existing dependency from the package manifest.

Synopsis

buffrs remove <PACKAGE>

Description

The remove command is the recommended way to remove a dependency (identified by package name) from the manifest. It modifies the manifest and will produce an error if the specified package cannot be found. It implements the opposite operation from the add command.

Lockfile Interaction

Currently removing a dependency won't automatically update the lockfile (Proto.lock). This is planned to change, but for now make sure to follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

buffrs lock

Commands for working with the Buffrs lockfile (Proto.lock).

Synopsis

buffrs lock <COMMAND>

Description

The lock command is a subcommand group for interacting with the Buffrs lockfile. The lockfile (Proto.lock) records the exact resolved versions of all dependencies and is managed automatically by buffrs install.

Subcommands

buffrs lock print-files – Print the locked file requirements as JSON to stdout.

buffrs lock print-files

Prints the locked files as JSON to stdout.

Synopsis

buffrs lock print-files

Description

Note: This command is designed for consumption through other scripts and programs.

Using this command you can retrieve a list of files that buffrs downloads according to the lockfile. For correct behavior please make sure your Proto.lock is up to date when using this command!

Example

Given a project that depends on a physics package at version 1.0.0 and a populated Proto.lock:

Running buffrs lock print-files will print the following output derived from the lockfile:

[
  {
    "url": "https://your.internal.registry/artifactory/your-repository/physics/physics-1.0.0.tgz",
    "digest": "sha256:61ecdcd949c7b234160dc5aacb4546a21512de4ff8ea85f2fdd7d5fff2bf92b5"
  }
]

This way you can programmatically consume this (e.g. in nix, bash, etc) and download the files if your project while maintaining integrity.

Package Commands

Package commands manage a local package, and are responsible for initializing the project structure, installing and uninstalling dependencies and building a publishable artifact from the current package state.

Index

buffrs init

Initializes the current directory as a Buffrs project.

Synopsis

buffrs init [name]

buffrs init --lib [name]

buffrs init --api [name]

Description

This command prepares the current directory as a Buffrs project, by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories.

By default, if no name is given, the current directory name is used as the package name. Note that there are special constraints on valid package names (see Package Name Specification for more details).

By default, if no package type is provided, impl (implementation) will be used. The meaning of this is described in Package Types.

buffrs new

Initializes a Buffrs project in a new folder created in the current directory.

Synopsis

buffrs new --lib <NAME>

buffrs new --api <NAME>

Description

This command creates a new Buffrs project with the provided name by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories in a new directory created at the current location.

A package type (--lib or --api) must be provided. The meaning of each type is described in Package Types.

Unlike buffrs init which initializes the current directory, this command creates a new subdirectory named after the package.

buffrs lint

Lints your protocol buffers for the (Buffrs Protocol Buffer Rules)

Synopsis

buffrs lint

Description

This command lints your local package (defined in proto/*.proto) for a set of rules defined in the (Buffrs Protocol Buffer Rules). They contain a set of rules ranging from style to package layout (like filenaming, package declaration etc.). This enables a common flavor to Buffrs packages which affect users.

One good example why this is required is the enforcement of euqality between the package declaration in the protocol buffers files (*.proto) and the Buffrs Package ID. This enables to expect that a Buffrs Package a declares the protocol buffer package a.* and prevents type colisions / ambiguity.

Example

Given a Buffrs Package abc that contains a protocol buffer file with the following file (proto/xyz.proto):

syntax = "proto3";

package xyz;

Executing buffrs lint would return a rule violation:

PackageName (https://helsing-ai.github.io/buffrs/rules/PackageName)

  × Make sure that the protobuf package name matches the buffer package name.
  ╰─▶   × package name is xyz but should have abc prefix

   ╭─[xyz.proto:1:1]
   ╰────
  help: Make sure the file name matches the package. For example, a package with the name `package.subpackage` should be stored in `proto/package/subpackage.proto`.

buffrs package

Generates a release tarball for the package in the current directory.

Synopsis

buffrs package

Options

--dry-run: prevents buffrs from actually writing the tarball to the filesystem
--output-directory: allows you to specify a directory to output the package
--set-version: allows you to override the version set in the manifest
--preserve-mtime: preserve access time information when creating a package (defaults to false)

Description

Like the publish command, the package command bundles the package's protocol buffer files and manifest into a gzip-compressed tarball. However, unlike the publish command it does not actually interact with the registry, instead it only writes the release tarball into the current directory. This is useful for manual distribution and for safely validating the package setup.

buffrs install

Downloads and installs dependencies specified in the manifest.

Synopsis

buffrs install [--offline] [--preserve-local-mtime]

Description

This command manages Buffrs local set of installed dependency packages. It is meant to be run from the root of the Buffrs project, where the Proto.toml manifest file can be found. Currently, only API and implementation packages can have dependencies, so this command is only useful for those package types.

The installation process will respect the requirements stated in the manifest file -- specifically, the version, registry and repository provided for each dependency. Each dependency may specify its own dependencies, via its manifest file, which will also be downloaded and its contents unpacked flatly to the local filesystem, under the shared proto/vendor path prefix (see Project Layout for more information). Only one version of each package can be installed, so if there is a conflicting requirement, installation will fail.

Since version 0.12, buffrs install supports local dependencies and workspaces:

Local dependencies: Dependencies specified with path = "..." are recursively resolved and installed
Workspaces: When run from a workspace root, installs dependencies for all workspace members

Once installation has completed, the resolved packages versions will be frozen and captured in a Proto.lock file, which ensures that future installations (local or performed in another machine) will install the exact same dependency versions. This file is managed automatically and should be kept under version control, so that others can reproduce your local installation.

Lockfile

The install command manages the Buffrs lockfile (Proto.lock) automatically. If one doesn't exist when the command is invoked, one is created after the installation has completed.

If dependencies have been added or removed since the last invocation, the lockfile will be modified accordingly. If the manifest requirements conflict with the lockfile (i.e. the manifest requests a different version than the one that was locked), installation will fail.

Versions are locked upon first installation, and will persist until the lockfile is regenerated with buffrs lock, dependencies are explicitly upgraded via buffrs update (or a manual edit of the manifest) or they have been removed. Once removed, if dependencies are added back again, a different version may be automatically selected and locked.

Transitive dependencies

Transitive dependencies are also managed by the current project's lockfile. Even if dependencies provide their own lockfile, those won't be used.

Options

`--offline`

Do not make any network requests. All packages must already be available in the local cache. If a package would need to be downloaded, the command fails with a human-readable error suggesting how to populate the cache.

This is useful for reproducible builds in sandboxed environments (e.g. Nix) where network access is not available. In such setups, the cache can be pre-populated using the BUFFRS_CACHE environment variable (see Environment Variables).

`--preserve-local-mtime`

Preserve access time information when installing a local dependency. Defaults to true.

buffrs uninstall

Deletes all installed dependencies from the local filesystem.

Synopsis

buffrs uninstall

Description

This command does the reverse operation from the install command, and will clear out the proto/vendors directory, thus removing all installed dependencies from the local filesystem. This is generally safe to do as the vendors directory is managed by Buffrs and shouldn't contain any custom proto files. Subsequently invoking the install command should restore the exact same files, assuming the lockfile hasn't been regenerated.

Since version 0.12, when run from a workspace root, this command will clear vendor directories for all workspace members.

buffrs list

Lists all protobuf files (.proto) managed by Buffrs to standard out.

Synopsis

buffrs list|ls

Description

This command lists all protobuf files managed by Buffrs. This way the output can be fed dynamically into external code generation tools like protoc.

Example

Given a project that depends on a physics package (that provides two .proto files: temperature.proto and mass.proto). Once it's dependencies are installed, the structure of the filesystem would look similar to this:

.
├── Proto.toml
└── proto
    ├── some.proto
    └── vendor
        └── physics
            ├── Proto.toml
            ├── temperature.proto
            └── mass.proto

Using buffrs ls you can feed the installed protocol buffer files of all package dynamically into another command line tool like protoc to generate code, or run lints:

protoc --cpp_out ./cpp --include proto $(buffrs ls)

The raw output of buffrs ls would return the following three paths:

proto/some.proto proto/vendor/physics/temperature.proto proto/vendor/physics/mass.proto

Publishing Commands

Publish commands interface with a remote registry and are primarily responsible for managing release publications.

Also in this category are commands to manage locally saved registry credentials.

Index

Saves an authentication token in the credentials store.

Synopsis

buffrs login --registry <url>

Description

This command prompts for an API or Identity token that can be used to authenticate with Artifactory for downloading and publishing packages.

The token is currently stored in $HOME/.buffrs/credentials.toml in the following format:

[[credentials]]
uri = "https://example.com/artifactory"
token = "<secret>"

buffrs logout

Removes an authentication token from the credentials store.

Synopsis

buffrs logout --registry <url>

Description

This command removes a previously saved token from the credentials store by its associated registry URL. Future invocations of publish and install that involve the given registry will then default to unauthenticated mode.

The credentials are currently stored in $HOME/.buffrs/credentials.toml.

buffrs publish

Generates a release and publishes it to the specified registry.

Synopsis

buffrs publish [OPTIONS] --registry <REGISTRY> --repository <REPOSITORY>

Options

--allow-dirty: allows publishing the package even if the repository has uncommitted changes.
--dry-run: causes a release bundle to be generated but skips uploading to the registry.
--set-version: allows you to override the version set in the manifest
--preserve-mtime: preserve access time information when creating a package (defaults to true)

Description

The publish command bundles the package's protocol buffer files and manifest into a gzip-compressed tarball, which is then uploaded to the specified registry and repository for publication. Once published the artifact will be available for other packages to be installed as dependencies.

In order for this command to be successful, the registry must be reachable via the network, and if authorization is required, credentials must have been previously saved via a buffrs login invocation.

By default, Buffrs does not allow publishing packages from git repositories in a dirty state (note: this requires the git feature to be enabled). This behaviour can be overridden by passing the --allow-dirty flag.

Since version 0.12, buffrs publish supports local dependencies and workspaces:

Local dependencies: Automatically publishes local dependencies first, then rewrites references to point to the registry
Workspaces: When run from a workspace root, publishes all workspace members in dependency order with automatic deduplication

Supported project types

Only Buffrs libraries and API packages can be packaged and published. More details in Package Types.

Library packages cannot have dependencies, so releasing this kind of package may fail if any are provided in the manifest. API dependencies on library packages is also forbidden and will cause publication to fail.

FAQ

We recently expanded the capabilities of Buffrs a bit and made it so it can handle being connected to multiple registries. For this reason, you'll have to add --registry http://my-registry.jfrog.io/artifactory to all three.

Note that buffrs login had a --url flag previously. It was renamed to --registry for the sake of consistency.

Why is my `credentials.toml` file broken?

Because we expanded Buffrs and made it capable of connecting to multiple registries, we had to make some changes to how we store our credentials.

When it only supported a single registry, it looked like this:

[artifactory]
url = "https://org.jfrog.io/artifactory"
password = "some-token"

And now it looks like this, supporting multiple regisitries:

[[credentials]]
uri = "https://org1.jfrog.io/artifactory"
token = "some-token"

[[credentials]]
uri = "https://org2.jfrog.io/artifactory"
token = "some-other-token"

Why can't I log in with a username?

buffrs login no longer supports the --username flag, as we no longer use BasicAuth. Instead we set the Authorization header which enables support for identity tokens, jwt, and encoded basic auth tokens at the same time.