The Buffrs Book

Buffrs is a protocol buffers package manager. Buffrs downloads your projects protocol buffer dependencies, creates distributable packages for libraries and apis, and uploads them to a package registry.

Sections

Getting Started

To get started with Buffrs, install it and set up your first package.

Buffrs Guide

An in-depth guide on how to use Buffrs to distribute protocol buffers.

Buffrs Integrations

Learn how to integrate Buffrs into existing build tools for a seamless development experience.

Buffrs Reference

Detailed aspects of Buffrs behavior, including advanced configuration and troubleshooting resources.

Buffrs Registry Reference

Advanced resources on the Buffrs Registry.

Buffrs Commands

Detailed documentation of the commands of the Buffrs CLI.

Getting Started

To get started with buffrs, install it and set up your first package.

• Installation
• First steps with Buffrs

Installation

Install Buffrs

The easiest way to get buffrs is to install the current stable release from crates.io using:

cargo install buffrs

As of right now you are required to authenticate yourself against your private artifactory instance (which will be replaced by the Buffrs Registry in Q4 2023).

Logging in to your instance is done using the following buffrs command:

buffrs login --registry https://<organization>.jfrog.io/artifactory

You will be prompted for an artifactory identity token which you can create within the artifactory user interface or programmatically through terraform.

Build and Install Buffrs from Source

As alternative installation method you can clone the Buffrs Repository and install it locally using Cargo (cargo install --path .).

First Steps with Buffrs

This section gives you a quick intro to the buffrs command line interface. Let us take a look at its ability to declare protocol buffer dependencies and to publish new packages to the registry.

To initialize a new project with Buffrs, use buffrs init:

$ mkdir web-server && cd web-server
$ buffrs init --api

Note: By omitting the --api flag (or --lib flag respectively) you instruct Buffrs to not declare a local package and setup the project to be a consumer-only (e.g. a server implementation).

$ tree .
.
├── Proto.toml
└── proto
    └── vendor

2 directories, 1 file

This is all we need to get started. Now let’s check out the newly created Proto.toml:

[package]
name = "web-server"
version = "0.1.0"
type = "api"

[dependencies]

This is called a Buffrs Manifest, and it contains all of the metadata that Buffrs needs to know about your package to install dependencies and distribute your protocol buffers as a package.

Let us define a dependency of the webserver on a hypothetical library called user in the datatypes repository.

This is done by invoking buffrs add:

$ buffrs add --registry https://your.registry.com datatypes/user@=0.1.0

The result is a dependency in the Proto.toml:

[dependencies.user]
version = "=0.1.0"
repository = "datatypes"
registry = "https://your.registry.com/"

Going further

For more details on using Buffrs, check out the Buffrs Guide

Buffrs Guide

This guide will give you all that you need to know about how to use Buffrs to develop and consume protocol buffer packages.

• Why Buffrs Exists
• The Framework
• Package Types
• Dependencies
• Creating a Package
• Consuming Packages
• Local Dependencies
• Import System
• Project Layout
• Manifest vs Lockfile
• Continuous Integration
• Buffrs Home

Why Buffrs Exists

Modern gRPC based software platforms make extensive use of protocol buffers to define and implement inter-service APIs. While the open-source protocol buffers ecosystem makes it very easy to get started with a few services and a handful of messages, recurring difficulties occur with scaling gRPC APIs beyond immediate repository boundaries for two main reasons:

1. Limited tooling for packaging, publishing, and distributing protocol buffer definitions.
2. The lack of widely-adopted best practices for API decomposition and reuse.

To overcome these problems we developed Buffrs - a package manager for protocol buffers. Using Buffrs, engineers package protocol buffer definitions, publish them to a shared registry, and integrate them seamlessly into their projects using versioned API dependencies and batteries-included build tool integration.

For a detailed differentiation against existing approaches (like buf, bazel and git submodules) and architecture deep dive take a look at the announcement post.

The Framework

To understand the distribution and decomposition framework that Buffrs provides it is useful to understand which properties of API management systems are desirable and why. Key aspects include:

Versioning

A versioning scheme for APIs — similar to versioned library dependencies — explicitly encodes compatibility properties. This allows developers to make either backwards-compatible or breaking changes and encode the compatibility guarantees in the API version: a minor version upgrade “just works”, while a new major version API may require manual migration/adaption in the consuming server/client implementations.

Source Compatibility

Given versioned protocol buffer APIs with explicit compatibility guarantees, it is desirable to have a system in which wire-format compatible APIs are also source-code compatible. This means that engineers can update minor patches automatically and that APIs that build upon the same protocol buffer types can be used with the same generated code types. This is especially important in strict languages like Rust (due to, e.g., the orphan rule).

Composition

Enabling engineers to reuse and combine code in order to build new systems from existing building blocks is a key feature. A common composition scheme with protocol buffers is to use a set of base messages and data types across many APIs.

Discoverability

Before engineers can reuse and compose protocol buffers, they need to be able to discover and understand what APIs exist and how to use them. Discoverability is a significant accelerator for engineering productivity and helps developers stay abreast of the evolution of APIs and architecture.

Protocol Buffers as a First Class Citizen

Buffrs decided to take a unique approach compared to other protocol buffer management systems (or systems capable of distributing protocol buffers) like buf and bazel. Buffrs is essentially a package manager for protocol buffers. Protocol buffers are treated as a first class citizen within Buffrs – which means that they are seen as distributable units called packages.

Complex software projects frequently turn out to depend on different versions of the same APIs over time and individual components in those systems may have diverging compatibility guarantees. Internal services may break their API backwards compatibility, way more frequent than external gateways that serve millions of users.

The three fundamental ideas of Buffrs to enable a stable management approach for scenarios like the above are:

Buffrs Packages

A closed and meaningful unit of protocol buffers which enables either productivity through shared types (e.g., Google's google.protobuf.Timestamp) or describes a domain / API (e.g., service GeoLocator).

Buffrs Registry

A central registry for managing and hosting packages, documentation, enabling engineers to search and find and to implement access control.

Local Code Generation

The last block is local code generation. This enables projects to freely diverge in their actual implementations by choosing a code generation implementation which fits their specific needs while still maintaining complete wire compatibility. This prevents language lock-ins and makes any version of any package usable for any language that exists today or might exist in the future (given it has a protobuf code generator).

Package Types

Buffrs makes distinctions between two different packages:

[package]
type = "lib" | "api"

This is used in order to fuel composition and type reuse across APIs and thus enable shared types + wire compatibility.

lib – Libraries

Libraries contain atomic and composite type definitions that describe a domain. (e.g. physics, auth, time etc.). This pattern is really useful for scaling systems and maintaining dozens of APIs as they can share types and thus get the aforementioned benefits of source code and wire compatibility for free.

An example of a proto library named time that depends on google:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", ... }

syntax = "proto3";

package time;

import "google/timestamp.proto";

/// A timestamp wrapper for various formats
message Time {
  oneof format {
    string rfc3339 = 1;
    uint64 unix = 2;
    google.protobuf.Timestamp google = 3;
    ..
  }
}

api – APIs

APIs are the next logical building block for real world systems – they define services and RPCs that your server can implement. You can use the aforementioned libraries to fuel your development / API definition experience.

A good example of an API could be an imaginary logging service that makes use of the just declared time.Time:

[package]
name = "logging"
type = "api"
version = "1.0.0"

[dependencies]
time = { version = "=1.0.0", ... }

syntax = "proto3";

package logging;

import "time/time.proto";

service Logging {
  rpc critical(LogInput) returns (LogOutput);
  rpc telemetry(LogInput) returns (LogOutput);
  rpc healthiness(HealthInput) returns (HealthOutput);
}

message LogInput { string context = 1; time.Time timestamp = 2; }
message LogOutput { }

message HealthInput { bool db = 1; }
message HealthOutput { }

Creating a Package

Creating a new, consumable, Buffrs package can be done in four steps:

1. Initialize a project using buffrs init
2. Set metadata, package type, version correctly and declare the dependencies that you want to use.
3. Declare message, enum and service types you want to publish in the newly created proto folder using .proto files.
4. Publish to a registry using buffrs publish.

Example: Publishing physics

Initialize Your Project

Start by initializing a new Buffrs project using the buffrs init command. This will set up the basic structure for your package and allow you to manage your package's metadata and dependencies.

mkdir physics
cd physics
buffrs init --lib

Define Package Metadata and Dependencies

In your project folder, locate the Proto.toml file. This is where you'll specify your package's metadata and dependencies. Open this file in a text editor.

Here's an example Proto.toml file:

# Package metadata
[package]
package = "physics"
version = "1.0.0"
type = "lib"
description = "A library containing physic related types"

# Declare dependencies (none in this case)
[dependencies]

Define Message, Enum, and Service Types

Inside your project directory, buffrs init created a proto folder. This is where you will store your .proto files that define your library or api.

An example file structure:

physics
├── Proto.toml
└── proto
    ├── temperature.proto
    ├── mass.proto
    └── vendor

Write your Protocol Buffer definitions in these .proto files. Here's a simple example of the temperature.proto file that could be in a physics library:

syntax = "proto3";

package physics.temperature;

// Define temperature units
enum TemperatureUnit {
  CELSIUS = 0;
  FAHRENHEIT = 1;
  KELVIN = 2;
}

// Define a message for temperature
message Temperature {
  double value = 1;            // Temperature value
  TemperatureUnit unit = 2;    // Temperature unit (Celsius, Fahrenheit, Kelvin)
}

Publish Your Package

Once you've set up your Buffrs package and defined your Protocol Buffer definitions, you can publish it to a registry using the buffrs publish command. Make sure you're logged in to the registry if required.

buffrs publish --registry https://your.registry.com --repository tutorial

Your package will be uploaded to the registry, and others can now consume it using Buffrs.

Congratulations! You've successfully published your Buffrs package. Other developers can now use your Protocol Buffer definitions by adding your package as a dependency in their Buffrs projects.

That's it! You've created a Buffrs package that others can easily consume. Remember to keep your package up-to-date and well-documented to make it even more valuable to the community.

Consuming Packages

As described in the package types section you can declare dependencies through your project's Proto.toml. This is true for both libraries and APIs. But what if you want to implement your server and you need to consume buffrs packages that contain your type and service definitions?

So there are three scenarios in which you would want to depend on other packages:

a) You are defining a library and you want to make use of an external type coming from another library (e.g. google for basic types such as google.None). b) You are defining an API and you want to use libraries to reuse common types for that domain (e.g. time or physics) c) You are implementing your server and you want to get access to API definitions to generate bindings.

The good news are: They are all achieved in a similar fashion. You make use of the [dependencies] key in your manifest to declare the packages that your projects needs – either to publish another package or to compile to protos to bindings.

Examples

Libraries & APIs

This section is identical for libraries and APIs.

An example of the time library reusing the google library:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository> }

Running buffrs install yields you with the following filesystem:

time
├── Proto.toml
└── proto
    ├── time.proto
    └── vendor
        ├── time
        ├   └── ..
        └── google
            ├── any.proto
            ├── ..
            ├── struct.proto
            └── timestamp.proto

You can now develop your library and publish it using buffrs publish.

Servers

If you want to implement your server and thus use e.g. a logging API the only major difference is the lack of the [package] section in your manifest.

[dependencies]
logging = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository> }

Running a buffrs install yields you the very same as above, except for the omitted local package and the logging dependency instead of time.

.
├── Proto.toml
└── proto
    └── vendor
        └── logging
            └── logger.proto

Import System

To reuse already defined messages, protobuf files can be imported from both dependencies and the managed package itself. Unique identification of the files is made available through the name of the package as declared in Proto.toml which is used as root of the imports.

For a dependency units:

units
├── Proto.toml
├── weight.proto
└── length.proto

and a package named physic:

physic
├── Proto.toml
└── proto
    ├── root.proto
    ├── length.proto
    └── calculations
        ├── distance.proto
        └── graph.proto

messages can be imported from both packages relative to their root:

// root.proto
syntax = "proto3";

package physic;

import "physic/length.proto";
import "physic/calculations/distance.proto";
import "units/length.proto";

message Lengths {
    units.Meter meter = 1;
    physic.Parsec parsec = 2;
    physic.calculations.Haversine = 3;
}

Project Layout

To get an understanding of the project layout that buffers uses it is helpful to start in a clean manner and introspect the outcome.

Lets create a new clean directory initialize for our physic library.

$ mkdir physic
$ cd physic
$ buffrs init --lib

This will initialize the following project structure:

physic
├── Proto.toml
└── proto
    └── vendor

This will create the Proto.toml file which is the manifest file that buffrs uses. The proto directory, which is the source directory for all your protocol buffer definitions and the proto/vendor directory, which contains external protocol buffers.

Important: The vendor directory is managed by Buffrs, all manual changes will be overridden / can cause unreproducible behavior.

Manifest (Proto.toml) vs Lockfile (Proto.lock)

Manifest – Proto.toml

Purpose: The Manifest (Proto.toml) serves as a specification file for the current package. It includes metadata, dependencies, and other package-related information.

Contents:

• Package Name: A unique name for the package.
• Version: Specifies the package version.
• Type: Specifies the type of this package (see Package Types).
• Description: Provides a brief description of the package.
• Metadata: Additional metadata like package category, tags, and any other relevant package details.
• Dependencies: Contains package names, version constraints, registry locations etc of the dependencies of the current package.

Usage: The Proto.toml is used to define the package's characteristics, metadata, and its dependencies, making it a comprehensive specification format. This file is included in compressed package artefacts and distributed alongside. For a usage guide see Creating A New Package.

Lockfile – Proto.lock

Purpose: The Lockfile (Proto.lock) is a separate, autogenerated and automanaged, file that records the exact versions of packages, including their transitive dependencies, which have been successfully resolved and installed.

Contents: It contains a detailed record of the package versions used, the registry or package source used during the installation and cryptographic hashes to ensure package integrity.

Usage: The lockfile is crucial for ensuring the reproducibility of installations. It guarantees that the same package versions are installed, regardless of changes in the upstream package registry, ensuring consistency across different installations.

Buffrs Home

$HOME/.buffrs

The buffrs home directory is a place for global configuration and data belonging to buffrs such as credentials and a cache. You should not need to interact with this folder manually.

The home directory that is used by buffrs can be configured via the BUFFRS_HOME environment variable. This enables you do override the default location in case you want to keep your home directory clean.

Continuous Integration

To utilize continuous integration for your Buffrs package (e.g. to automate code review and publishing of your packages) you can utilize the following templates for GitHub Actions and GitLab CI:

GitHub Actions

name: Buffrs

on:
  push:
    branches:
      - '*'
  tags:
    - '*'

env:
  REGISTRY: https://<org>.jfrog.io/artifactoy
  REPOSITORY: your-artifactory-repo

jobs:
  verify:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly

      - name: Set up Rust environment
        run: |
          rustup target add aarch64-unknown-linux-gnu
        shell: bash

      - name: Verify
        run: |
          cargo install --force buffrs
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs lint
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

  publish:
    runs-on: ubuntu-latest
    needs: build
    if: startsWith(github.ref, 'refs/tags/')

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Publish on tag
        run: |
          cargo install --force buffrs
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs publish --registry $REGISTRY --repository $REPOSITORY
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

GitLab CI

stages:
  - verify
  - publish

variables:
  TOKEN: $BUFFRS_TOKEN  # Your secret artifactory token
  REGISTRY: https://<org>.jfrog.io/artifactory
  REPOSITORY: your-artifactory-repo

verify:
  stage: verify
  script:
    - cargo install buffrs
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs lint
  only:
    - branches

publish:
  stage: publish
  script:
    - cargo install buffrs
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs publish --registry $REGISTRY --repository $REPOSITORY
  only:
    - tags

Buffrs Reference

This section covers detailed documentation on the Buffrs Ecosystem / Framework.

Index

• Editions
• Protocol Buffer Rules

Editions

Editions of buffrs mark a specific evolutionary state of the package manager. The edition system exists to allow for fast development of buffrs while allowing you to already migrate existing protobufs to buffrs even though it has not yet reached a stable version.

Editions can be either explicitly stated in the Proto.toml or are automatically inlined once a package is created using buffrs. This ensures that you dont need to care about them as a user but get the benefits.

Note: If you release a package with an edition that is incompatible with another one (e.g. if 0.7 is incompatible with 0.8) you will need to re-release the package for the new edition (by bumping the version, or overriding the existing package) to regain compatibility.

You may see errors like this if you try to consume (or work on) a package of another edition.

Error:   × could not deserialize Proto.toml
  ╰─▶ TOML parse error at line 1, column 1
        |
      1 | edition = "0.7"
        | ^^^^^^^^^^^^^^^
      unsupported manifest edition, supported editions of 0.8.0 are: 0.8

Canary Editions

edition = "0.7"

Canary editions are short-lived editions that are attached to a specific minor release of buffrs in the 0.x.x version range. The edition name contains the minor version it is usable for. E.g. the edition 0.7 is usable / supported by all 0.7.x buffrs releases. Compatibility beyond minor releases is not guaranteed as fundamental breaking changes may be introduced between editions.

Protocol Buffer Rules

This specification defines rules enforced by Buffrs to prevent package colisions, and provide uniformity and transparency for package consumers.

Rules with the 00XX code define the package and filesystem layout, where as rules with a 01XX code enforce certain protocol buffer definition rules.

0000 – Package Name Prefix / Symmetry

Enforces that the Buffrs Package ID is used as the prefix for all protocol buffer package declarations.

So given a Buffrs Package with the ID physics this enforces that the package only contains protocol buffer package declarations matching physics|physics.*;

A violation would cause type colisions and ambiguity when trying to resolve a type.

0010 – Sub-Package Declaration

Enforces that subpackages are declared through a sensible folder structure. Given a Buffrs Package with the ID physics the protocol buffer file that declares package physics.units; has to be called proto/units.proto.

Nested subpackages are represented / grouped through folders. So if one wants to declare package physics.units.temperature; the respective file must be located at proto/units/temperature.proto.

0020 – Root Package Declaration

Enforces that only one file at a time declares the root package.

Namely: If a Buffrs Package with the ID physics is defined, the proto/physics.proto must declare the the same package in the protocol buffer syntax through package physics;.

Buffrs Commands

This section covers detailed documentation on operating the Buffrs CLI. Some of the content may also apply to the library, particularly the command module.

The goal is to provide sufficient information to use the CLI effectively to manage a Buffrs project, also also capture context and behaviour that may otherwise be only expressed in code.

The help command should also provide useful information and is expected to generally be more up-to-date.

Index

• General Commands
  • buffrs
  • buffrs help
• Build Commands
  • buffrs list
• Manifest Commands
  • buffrs add
  • buffrs remove
• Package Commands
  • buffrs init
  • buffrs lint
  • buffrs package
  • buffrs install
  • buffrs uninstall
• Publishing Commands
  • buffrs login
  • buffrs logout
  • buffrs publish

General Commands

General commands interface with the CLI itself, in order to obtain access to built-in help or version information.

Index

• buffrs
• buffrs help

buffrs

The official Buffrs command-line interface.

Synopsis

buffrs

Description

When invoked without any arguments, the Buffrs binary defaults to printing out help information to the standard output.

This is equivalent to buffrs help, or invoking with the -h or --help flags.

Providing the -V or --version flags is also equivalent to buffrs version.

Output

Modern protobuf package management

Usage: buffrs <COMMAND>

Commands:
  init       Initializes a buffrs setup
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  list       Lists all protobuf files managed by Buffrs to stdout
  generate   Generate code from installed buffrs packages
  login      Logs you in for a registry
  logout     Logs you out from a registry
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

buffrs help

Prints out information about how to use the CLI.

Synopsis

buffrs help [command]

Description

When called by itself, this command lists all the supported commands along with a brief description.

When called with a command argument, it will provide specific help for that command.

Passing the -h or --help flags is equivalent to invoking this command.

Examples

> buffrs help
Modern protobuf package management

Usage: buffrs <COMMAND>

Commands:
  init       Initializes a buffrs setup
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  generate   Generate code from installed buffrs packages
  login      Logs you in for a registry
  logout     Logs you out from a registry
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

> buffrs help init
Initializes a buffrs setup

Usage: buffrs init [OPTIONS] [PACKAGE]

Arguments:
  [PACKAGE]  The package name used for initialization

Options:
      --lib      Sets up the package as lib
      --api      Sets up the package as api
  -h, --help     Print help
  -V, --version  Print version

Build Commands

Build commands assist implementation projects with generating code and documentation from Buffrs-managed protocol buffer files.

Index

• buffrs list

buffrs generate

buffrs list

Lists all protobuf files (.proto) managed by Buffrs to standard out.

Synopsis

buffrs list|ls

Description

This command lists all protobuf files managed by Buffrs. This way the output can be fed dynamically into external code generation tools like protoc.

Example

Given a project that depends on a physics package (that provides two .proto files: temperature.proto and mass.proto). Once it's dependencies are installed, the structure of the filesystem would look similar to this:

.
├── Proto.toml
└── proto
    ├── some.proto
    └── vendor
        └── physics
            ├── Proto.toml
            ├── temperature.proto
            └── mass.proto

Using buffrs ls you can feed the installed protocol buffer files of all package dynamically into another command line tool like protoc to generate code, or run lints:

protoc --cpp_out ./cpp --include proto $(buffrs ls)

The raw output of buffrs ls would return the following three paths:

proto/some.proto proto/vendor/physics/temperature.proto proto/vendor/physics/mass.proto

Manifest Commands

Manifest commands manage the package's manifest file, which declares important metadata like the package name, version and dependencies.

These commands can be used to include, remove and update dependencies while keeping the lockfile in sync.

Index

• buffrs add
• buffrs remove

buffrs add

Adds a new dependency to the package manifest.

Synopsis

buffrs add --registry <REGISTRY> <DEPENDENCY>

Description

The add command is the recommended way to include a new dependency in the current package. It modifies the local manifest file and will overwrite a pre-existing entry for the same dependency package if it exists.

Dependency locator format

The dependency should be specified with the repository, name and version according to the following format:

<repository>/<package>[@<version>]

Note: the version can be omitted (or set to @latest), in which case it will default to the latest version of this artifact in the registry.

The repository name should adhere to lower-kebab case (e.g. my-buffrs-repo). The package name has its own set of constraints as detailed in Package Name Specification. When specified, the version must adhere to the Semantic Version convention (e.g. 1.2.3) -- see SemVer compatibility for more information.

Currently there is no support for resolving version operators but the specific version has to be provided. This means ^1.0.0, <2.3.0, ~2.0.0, etc. can't be installed, but =1.2.3 has to be provided.

Lockfile interaction

Currently adding a new dependency won't automatically update the lockfile (Proto.lock). This is planned to change, but for now follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

buffrs remove

Removes an existing dependency from the package manifest.

Synopsis

buffrs remove <PACKAGE>

Description

The remove command is the recommended way to remove a dependency (identified by package name) from the manifest. It modifies the manifest and will produce an error if the specified package cannot be found. It implements the opposite operation from the add command.

Lockfile Interaction

Currently removing a dependency won't automatically update the lockfile (Proto.lock). This is planned to change, but for now make sure to follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

Package Commands

Package commands manage a local package, and are responsible for initializing the project structure, installing and uninstalling dependencies and building a publishable artifact from the current package state.

Index

• buffrs init
• buffrs lint
• buffrs package
• buffrs install
• buffrs uninstall

buffrs init

Initializes the current directory as a Buffrs project.

Synopsis

buffrs init [name]

buffrs init --lib [name]

buffrs init --api [name]

Description

This command prepares the current directory as a Buffrs project, by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories.

By default, if no name is given, the current directory name is used as the package name. Note that there are special constraints on valid package names (see Package Name Specification for more details).

By default, if no package type is provided, impl (implementation) will be used. The meaning of this is described in Package Types.

buffrs init

Initializes a Buffrs project in a new folder created in the current directory.

Synopsis

buffrs new <NAME>

buffrs new --lib <NAME>

buffrs new --api <NAME>

Description

This command creates a new Buffrs project with the provided name by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories in a new directory created at the current location.

By default, if no package type is provided, impl (implementation) will be used. The meaning of this is described in Package Types.

buffrs lint

Lints your protocol buffers for the (Buffrs Protocol Buffer Rules)

Synopsis

buffrs lint

Description

This command lints your local package (defined in proto/*.proto) for a set of rules defined in the (Buffrs Protocol Buffer Rules). They contain a set of rules ranging from style to package layout (like filenaming, package declaration etc.). This enables a common flavor to Buffrs packages which affect users.

One good example why this is required is the enforcement of euqality between the package declaration in the protocol buffers files (*.proto) and the Buffrs Package ID. This enables to expect that a Buffrs Package a declares the protocol buffer package a.* and prevents type colisions / ambiguity.

Example

Given a Buffrs Package abc that contains a protocol buffer file with the following file (proto/xyz.proto):

syntax = "proto3";

package xyz;

Executing buffrs lint would return a rule violation:

PackageName (https://helsing-ai.github.io/buffrs/rules/PackageName)

  × Make sure that the protobuf package name matches the buffer package name.
  ╰─▶   × package name is xyz but should have abc prefix

   ╭─[xyz.proto:1:1]
   ╰────
  help: Make sure the file name matches the package. For example, a package with the name `package.subpackage` should be stored in `proto/package/subpackage.proto`.

buffrs package

Generates a release tarball for the package in the current directory.

Synopsis

buffrs package

Options

• --dry-run: prevents buffrs from actually writing the tarball to the filesystem
• --output-directory: allows you to specify a directory to output the package
• --set-version: allows you to override the version set in the manifest

Description

Like the publish command, the package command bundles the package's protocol buffer files and manifest into a gzip-compressed tarball. However, unlike the publish command it does not actually interact with the registry, instead it only writes the release tarball into the current directory. This is useful for manual distribution and for safely validating the package setup.

buffrs install

Downloads and installs dependencies specified in the manifest.

Synopsis

buffrs install

Description

This command manages Buffrs local set of installed dependency packages. It is meant to be run from the root of the Buffrs project, where the Proto.toml manifest file can be found. Currently, only API and implementation packages can have dependencies, so this command is only useful for those package types.

The installation process will respect the requirements stated in the manifest file -- specifically, the version, registry and repository provided for each dependency. Each dependency may specify its own dependencies, via its manifest file, which will also be downloaded and its contents unpacked flatly to the local filesystem, under the shared proto/vendor path prefix (see Project Layout for more information). Only one version of each package can be installed, so if there is a conflicting requirement, installation will fail.

Once installation has completed, the resolved packages versions will be frozen and captured in a Proto.lock file, which ensures that future installations (local or performed in another machine) will install the exact same dependency versions. This file is managed automatically and should be kept under version control, so that others can reproduce your local installation.

Lockfile

The install command manages the Buffrs lockfile (Proto.lock) automatically. If one doesn't exist when the command is invoked, one is created after the installation has completed.

If dependencies have been added or removed since the last invocation, the lockfile will be modified accordingly. If the manifest requirements conflict with the lockfile (i.e. the manifest requests a different version than the one that was locked), installation will fail.

Versions are locked upon first installation, and will persist until the lockfile is regenerated with buffrs lock, dependencies are explicitly upgraded via buffrs update (or a manual edit of the manifest) or they have been removed. Once removed, if dependencies are added back again, a different version may be automatically selected and locked.

Transitive dependencies

Transitive dependencies are also managed by the current project's lockfile. Even if dependencies provide their own lockfile, those won't be used.

buffrs uninstall

Deletes all installed dependencies from the local filesystem.

Synopsis

buffrs uninstall

Description

This command does the reverse operation from the install command, and will clear out the proto/vendors directory, thus removing all installed dependencies from the local filesystem. This is generally safe to do as the vendors directory is managed by Buffrs and shouldn't contain any custom proto files. Subsequently invoking the install command should restore the exact same files, assuming the lockfile hasn't been regenerated.

Publishing Commands

Publish commands interface with a remote registry and are primarily responsible for managing release publications.

Also in this category are commands to manage locally saved registry credentials.

Index

• buffrs login
• buffrs logout
• buffrs publish

buffrs login

Saves an authentication token in the credentials store.

Synopsis

buffrs login --registry <url>

Description

This command prompts for an API or Identity token that can be used to authenticate with Artifactory for downloading and publishing packages.

The token is currently stored in $HOME/.buffrs/credentials.toml in the following format:

[[credentials]]
uri = "https://example.com/artifactory"
token = "<secret>"

buffrs logout

Removes an authentication token from the credentials store.

Synopsis

buffrs logout --registry <url>

Description

This command removes a previously saved token from the credentials store by its associated registry URL. Future invocations of publish and install that involve the given registry will then default to unauthenticated mode.

The credentials are currently stored in $HOME/.buffrs/credentials.toml.

buffrs publish

Generates a release and publishes it to the specified registry.

Synopsis

buffrs publish [OPTIONS] --registry <REGISTRY> --repository <REPOSITORY>

Options

• --allow-dirty: allows publishing the package even if the repository has uncommitted changes.
• --dry-run: causes a release bundle to be generated but skips uploading to the registry.
• --set-version: allows you to override the version set in the manifest

Description

The publish command bundles the package's protocol buffer files and manifest into a gzip-compressed tarball, which is then uploaded to the specified registry and repository for publication. Once published the artifact will be available for other packages to be installed as dependencies.

In order for this command to be successful, the registry must be reachable via the network, and if authorization is required, credentials must have been previously saved via a buffrs login invocation.

By default, Buffrs does not allow publishing packages from git repositories in a dirty state (note: this requires the git feature to be enabled). This behaviour can be overridden by passing the --allow-dirty flag.

Supported project types

Only Buffrs libraries and API packages can be packaged and published. More details in Package Types.

Library packages cannot have dependencies, so releasing this kind of package may fail if any are provided in the manifest. API dependencies on library packages is also forbidden and will cause publication to fail.

FAQ

Why doesn't buffrs add, buffrs publish, or buffrs login work anymore?

We recently expanded the capabilities of Buffrs a bit and made it so it can handle being connected to multiple registries. For this reason, you'll have to add --registry http://my-registry.jfrog.io/artifactory to all three.

Note that buffrs login had a --url flag previously. It was renamed to --registry for the sake of consistency.

Why is my credentials.toml file broken?

Because we expanded Buffrs and made it capable of connecting to multiple registries, we had to make some changes to how we store our credentials.

When it only supported a single registry, it looked like this:

[artifactory]
url = "https://org.jfrog.io/artifactory"
password = "some-token"

And now it looks like this, supporting multiple regisitries:

[[credentials]]
uri = "https://org1.jfrog.io/artifactory"
token = "some-token"

[[credentials]]
uri = "https://org2.jfrog.io/artifactory"
token = "some-other-token"

Why can't I log in with a username?

buffrs login no longer supports the --username flag, as we no longer use BasicAuth. Instead we set the Authorization header which enables support for identity tokens, jwt, and encoded basic auth tokens at the same time.