Martin Pitt and Daan De Meyer introduce tine: reproducible builds of RPM, Go, and Rust components into bootable images from minimal host requirements.

From Thin Air to Bootable Images: The tine Build System

By Daan De Meyer and Martin Pitt

This post is part of a series covering some of the open source work we have been doing in recent months. Today we introduce and publish tine, our new Buck2-based build system.

Our Requirements

Building an operating system with cryptographically verifiable integrity has to start with a build system with these very properties. At the same time, we are part of the greater open source community and want both to contribute and re-use as much existing work as possible. We also aim for efficient development with fast turnaround.

This roughly translates into the following requirements for our build system:

Minimal host requirements. It must be self-contained and must minimize external dependencies, so that it can be run in any environment.
Full control over inputs. It must allow pinning every piece of software that goes into a product. It must support a choice of upstream distributions (Fedora, CentOS, Arch, Debian, etc.) and reuse existing packages where possible, while still making it easy to react quickly to CVEs and to diverge (either temporarily or permanently) from upstream packaging decisions when necessary.
Integrated package machinery. It must provide tooling for importing, updating, and merging imported packages.
Cheap world rebuilds. It must be able to rebuild the world on demand, e.g. after a gcc bump.
Hermetic, reproducible builds. All component and image builds must run in a hermetic environment and produce bitwise reproducible output.
Native image builds. It must be able to build bootable operating system images and systemd sysext images natively and concurrently.
Monorepo-based iteration. It must support maintaining the operating system in a single top-level monorepo for fast end-to-end iteration. A change to an imported rpm or to a Go or Rust component must be immediately buildable and testable across the full set of images, without intermediate commits or pushes and without elaborate version or dependency declarations.
First-class custom components. It must natively and efficiently build Go and Rust projects from pinned external repositories, for example kubernetes or varlink-http-bridge.
Scanner compatibility. Built images must work with standard SBOM tooling and security scanners such as syft/grype or trivy.
Caching. Builds must be able to retrieve unchanged components from a local and/or global cache. Building everything from scratch can take hours, and a developer is usually only working on a single component.

Existing Tools

Before building our own build tool, we evaluated several options.

mkosi

Given our team includes the creator and maintainer of mkosi, it was a natural first candidate to evaluate. But we quickly came to the conclusion that it has some fundamental shortcomings. It’s great at building individual images based on upstream packages, but becomes restrictive when building several weakly related images or if you need more control over the artifacts that make up an image. Building multiple images is limited to images that are intended to be shipped as part of the “main” image.

We need to build many different kinds of artifacts in a uniform and robust way, not just images. Hence the build needs to be orchestrated by a generic and flexible tool. The main build file should be a language calling into library functions like “compile a cargo crate” or “build a UKI”. mkosi is the opposite, it’s a framework: It knows how to build images, and only gives you free-form opaque hooks for the other kinds of builds. That leads to a bad experience when you want to build more than just images.

Open Build Service

The Open Build Service (OBS) is a powerful fully-integrated build system that is primarily used by SUSE and the openSUSE project to produce all of their artifacts, anything from packages to ISOs, and many other image formats. It has strong dependency tracking and supports a dizzying array of distributions.

However, it’s also the antithesis of “minimal host requirements”. The server side of it is required, central, and non-trivial to self-host. It’s also not a generic build system, meaning any new artifact types would either have to be modelled as packages or require heavy patches to OBS. We concluded that this lack of flexibility combined with its overall architecture would make it difficult for OBS to meet our requirements.

BuildStream

Apache BuildStream describes the operating system image as a graph of YAML "elements", each with its own sources, dependencies and build commands. BuildStream builds each one in a bubblewrap sandbox and caches the result under a hash of everything that went into it, similar to Buck2. It is mature and used to build freedesktop-sdk, GNOME OS, and WebKitGTK.

Our concerns with BuildStream are mostly around bootstrapping and extensibility. BuildStream is a Python application with compiled extensions and other dependencies. It relies on a separate set of helper programs, plus sandboxing tools from the host. Each of those can be pinned, but through different mechanisms, and even then the result still depends on the host's Python. In practice you run it from a pinned container image instead, but then you’re still dependent on an entire container runtime you don’t control.

BuildStream’s YAML is a plain data format, and is extended with Python plugins. YAML has no functions, so over time you end up copy-pasting across the project. Ultimately we decided to go for a tool with a better bootstrapping and pinning story as well as a more flexible language.

Antlir

Antlir is Meta's OS image builder, built on top of the Buck2 build system engine. Buck2 is Meta's open source build system with emphasis on correctness, flexibility, and caching as much as possible. Antlir implements various rules for building images with Buck2.

As Antlir is a high-level tool focused on Meta’s internal repository, it is naturally very opinionated and designed for Meta’s internal use cases. For example, it requires btrfs and is strongly focused on a single monorepo.

While we decided against using Antlir itself, its underlying engine, Buck2, turned out to be a good fit, and we ended up choosing it as the foundation for our own build system.

Our Build System: tine

In essence, tine is a set of opinionated Buck2 rules to build rpm, Rust crate and Go module components, UKIs, and images; it can sign images either with a hardware key through PKCS#11 or a locally generated key. The intent is to combine the best ideas from Antlir and mkosi into a single tool.

tine has only three requirements on its build host: git, python3 (just for its own bootstrapping, not for production builds) and user namespaces. From there, it bootstraps everything it needs from pinned declarations to get a reproducible and independent build environment. That can be a distribution as old or modern as you need.

An important tine concept is the “box”, which is a declared and pinned down environment to run a build task. Think containers or distrobox, but declared natively in Buck2’s language, and using Buck2’s caching and rebuild rules, so they build quickly and naturally, stay reproducible, and need no further dependencies to run. tine itself defines boxes for running rpmbuild, go, or cargo, or a bigger multi-purpose one called fedora.rawhide.box which contains e.g. systemd-ukify for building images and QEMU for running virtual machines. Your own project can define its own boxes.

tine’s Engine: Buck2

To better understand this post and the examples, here is a one-minute Buck2 primer for those familiar with Make or Meson:

Build file: A BUCK file is the directory's Makefile equivalent. It's written in a Python dialect called Starlark, and declares alltargets that you can build.
Cell: A named build graph root; these roughly follow the boundaries of git repositories: // is your own top-level project (the OS you want to build), tine// is the tine checkout which your project pulls in.
Target: A named node in the build graph, i.e. one particular thing that you want to build. They are addressed with an absolute path of the form cell//directory/sub:name, which refers to a target name defined in cell’s directory/sub/BUCK file. Within a cell, you can also use relative paths, likesubdir:name , or even just:name for a target in the current directory. A single target can publish several output variants (“subtargets”), e.g.:my_cool_os[qcow2] or:my_cool_os[sbom] .
Rule: The equivalent of a meson*_target() , or the structure of a Makefile rule: a Starlark expression which translates a target into a set ofactions and their parameters. It does not run anything by itself. For example, abootable_disk(name = “myos”, param1 = …) rule defines amyos target and invokes abootable_disk rule which translates it into actions like “install rpms”, “runsystemd-repart” and so on.
Action: One build command with its declared inputs and outputs, the equivalent of the commands in a Makefile rule. That abstraction allows running all of them consistently in a sandbox which only sees these inputs. When Starlark doesn’t suffice, these rules can be implemented with the full power of Python.

More information can be found on Buck2’s key concepts page.

Unlike Make or Meson, Buck2 never decides what to rebuild from timestamps. An action is keyed by a hash of all of its inputs: the sources, the tool binaries, the build platform configuration, and the command line itself. That is what makes its incremental builds correct and trustworthy, and it also allows taking an action's result from a shared cache instead of re-running it.

Walkthrough: Building a Bootable Image

Let’s walk through how to use tine in your own projects. We will build a very basic example from scratch: a bootable image based on Fedora Rawhide with a Go project, and boot it. In this example, we’ll use duf, a CLI tool that shows free/used disk space in a text terminal with nice ASCII art.

Let’s follow tine's README and set up a fresh demo git repository which pulls in tine and initializes it.

git init tine-demo
cd tine-demo
git submodule add https://github.com/amutable-systems/tine tine
tine/bin/tine init
git add .
git commit -m "initialize"

Now let’s add a BUCK file. We’ll walk through it in several blocks, but these all go in the same file. First we need to import some definitions. This is Starlark, so akin to Python’s import statements.

load("@tine//box:defs.bzl", "box")
load("@tine//git:defs.bzl", "git")
load("@tine//go:defs.bzl", "go")
load("@tine//image:defs.bzl", "image")

Next we need to define a build environment for the Go compiler. tine already offers a Fedora rawhide catalog. So, let’s just use that (hence the tine// cell) and Fedora’s golang package. A real project would likely define and track their parent OS catalog by itself, instead of blindly following tine’s.

box.new(
    name = "go.box",
    packages = ["golang"],
    release = "tine//catalog:fedora.rawhide.release",
)

Declare the duf Go project git repository which we want to build. tine requires pinning every input exactly, so we specify a git commit ID. That git repository is then passed as input to the go.package() rule which binds the above go.box and the git checkout, both referenced as relative targets (see above), hence the colon separator.

git.fetch(
    name = "duf.git",
    repo = "https://github.com/muesli/duf",
    rev = "4636deb4a7b707a9f04c602db033f9837e50b3f6",
)
go.package(
    name = "duf",
    box = ":go.box",  # a target in the current directory
    src = ":duf.git", # another target
)

With that we can already build and execute the binary.

tine/bin/tine buck run :duf
# [...]
# BUILD SUCCEEDED - starting your binary
# 5 local devices
# [...]

And now for the last big piece: the bootable image. Just as with the Go box, we re-use the tine catalog’s package manager that gets packages from Fedora Rawhide. This is the minimum set to be able to boot in a virtual machine, plus bash. As an extra ops (operation) this installs the built hello binary from the above go rule.

image.bootable_disk(
    name = "demo",
    package_manager = "tine//catalog:fedora.rawhide.package-manager",
    definitions = image.DEFAULT_USR_VERITY_PARTITIONS,
    version = "0.0.0",
    package_sets = ["bootable"],
    packages = ["bash"],
    ops = [
        image.copy(":duf[duf]", "/usr/bin/duf"),
    ],
)

We can ask Buck2 for all available build targets in a cell.

tine/bin/tine buck targets //...
# root//:demo
# root//:demo.initrd
# root//:duf
# root//:duf.git
# root//:example.git
# root//:go.box
# root//:go.box.exec

After all of that, we can now build the image. However, it’s far more interesting to actually see it live in QEMU. Let’s add a VM definition, with auto-login for convenience, that boots our shiny new demo image using QEMU and related tools from tine’s own Rawhide catalog.

image.vm(
    name = "demo-vm",
    autologin = "root",
    # re-using tine's rawhide box which has all of QEMU etc. installed
    box = "tine//catalog:fedora.rawhide.box",
    image = ":demo",
    credentials = {
        "firstboot.timezone": "UTC",
    }
)

This single command from a clean tree will then build the Go project, the image, and boot it.

tine/bin/tine buck run :demo-vm

You should see the following.

[  OK  ] Reached target graphical.target - Graphical Interface.
Fedora Linux 46 (Rawhide Prerelease)
Kernel 7.3.0-0.rc3.260916g9b87fdc9af2f.34.fc46.x86_64 on an x86_64 (hvc0)
fedora login: root (automatic login)
-bash-5.3# duf --help
Usage of duf:
      --all   include pseudo, duplicate, inaccessible file systems
[...]
-bash-5.3# ...
-bash-5.3# systemctl poweroff

Look at tine’s examples/ directory for BUCK and auxiliary files for various scenarios such as a SecureBoot/verity OS signed by either a generated or hardware key, how to build Go/Rust projects, the various kinds of rules and customizations which tine offers, or how to write integration tests.

SBOMs for Free

tine gives you a lot more for free. For example, it integrates the Syft SBOM generator so you can ask it to build a CycloneDX standard SBOM. Let’s also specify an output path, so that you don’t have to fish it out of buck-out/:

tine/bin/tine buck build --out /tmp/demo.cdx.json :demo[sbom][cyclonedx]

There’s Much More!

This blog post has only covered the basics. In order to get a fuller picture, please refer to the documentation. We think the following topics are the most helpful to get started.

The distro package machinery for importing, modifying, and syncing/merging packages from Fedora or Arch
Setting up a shared build cache, and the details of its threat model and design
Signing builds with hardware keys through PKCS#11
Including auditable cargo projects and Go projects into image builds
Using tine mount for rapid iteration on a built third-party git project component
tine comes with builtin tools for bumping dependencies and refreshing the catalog which you can re-use in your project to automate the regular housekeeping and get tested PRs for them, like this one
tine also supports Arch, BTW!

The next post in this series will be about the update and provisioning system we have built to distribute these and other images. We hope to see you then!

This post is part of a series covering some of the open source work we have been doing in recent months. Today we introduce and publish tine, our new Buck2-based build system.

Our Requirements

This roughly translates into the following requirements for our build system:

Minimal host requirements. It must be self-contained and must minimize external dependencies, so that it can be run in any environment.
Full control over inputs. It must allow pinning every piece of software that goes into a product. It must support a choice of upstream distributions (Fedora, CentOS, Arch, Debian, etc.) and reuse existing packages where possible, while still making it easy to react quickly to CVEs and to diverge (either temporarily or permanently) from upstream packaging decisions when necessary.
Integrated package machinery. It must provide tooling for importing, updating, and merging imported packages.
Cheap world rebuilds. It must be able to rebuild the world on demand, e.g. after a gcc bump.
Hermetic, reproducible builds. All component and image builds must run in a hermetic environment and produce bitwise reproducible output.
Native image builds. It must be able to build bootable operating system images and systemd sysext images natively and concurrently.
Monorepo-based iteration. It must support maintaining the operating system in a single top-level monorepo for fast end-to-end iteration. A change to an imported rpm or to a Go or Rust component must be immediately buildable and testable across the full set of images, without intermediate commits or pushes and without elaborate version or dependency declarations.
First-class custom components. It must natively and efficiently build Go and Rust projects from pinned external repositories, for example kubernetes or varlink-http-bridge.
Scanner compatibility. Built images must work with standard SBOM tooling and security scanners such as syft/grype or trivy.
Caching. Builds must be able to retrieve unchanged components from a local and/or global cache. Building everything from scratch can take hours, and a developer is usually only working on a single component.

Existing Tools

Before building our own build tool, we evaluated several options.

mkosi

Open Build Service

BuildStream

Antlir

While we decided against using Antlir itself, its underlying engine, Buck2, turned out to be a good fit, and we ended up choosing it as the foundation for our own build system.

Our Build System: tine

tine’s Engine: Buck2

To better understand this post and the examples, here is a one-minute Buck2 primer for those familiar with Make or Meson:

Build file: A BUCK file is the directory's Makefile equivalent. It's written in a Python dialect called Starlark, and declares alltargets that you can build.
Cell: A named build graph root; these roughly follow the boundaries of git repositories: // is your own top-level project (the OS you want to build), tine// is the tine checkout which your project pulls in.
Target: A named node in the build graph, i.e. one particular thing that you want to build. They are addressed with an absolute path of the form cell//directory/sub:name, which refers to a target name defined in cell’s directory/sub/BUCK file. Within a cell, you can also use relative paths, likesubdir:name , or even just:name for a target in the current directory. A single target can publish several output variants (“subtargets”), e.g.:my_cool_os[qcow2] or:my_cool_os[sbom] .
Rule: The equivalent of a meson*_target() , or the structure of a Makefile rule: a Starlark expression which translates a target into a set ofactions and their parameters. It does not run anything by itself. For example, abootable_disk(name = “myos”, param1 = …) rule defines amyos target and invokes abootable_disk rule which translates it into actions like “install rpms”, “runsystemd-repart” and so on.
Action: One build command with its declared inputs and outputs, the equivalent of the commands in a Makefile rule. That abstraction allows running all of them consistently in a sandbox which only sees these inputs. When Starlark doesn’t suffice, these rules can be implemented with the full power of Python.

More information can be found on Buck2’s key concepts page.

Walkthrough: Building a Bootable Image

Let’s follow tine's README and set up a fresh demo git repository which pulls in tine and initializes it.

git init tine-demo
cd tine-demo
git submodule add https://github.com/amutable-systems/tine tine
tine/bin/tine init
git add .
git commit -m "initialize"

load("@tine//box:defs.bzl", "box")
load("@tine//git:defs.bzl", "git")
load("@tine//go:defs.bzl", "go")
load("@tine//image:defs.bzl", "image")

box.new(
    name = "go.box",
    packages = ["golang"],
    release = "tine//catalog:fedora.rawhide.release",
)

git.fetch(
    name = "duf.git",
    repo = "https://github.com/muesli/duf",
    rev = "4636deb4a7b707a9f04c602db033f9837e50b3f6",
)
go.package(
    name = "duf",
    box = ":go.box",  # a target in the current directory
    src = ":duf.git", # another target
)

With that we can already build and execute the binary.

tine/bin/tine buck run :duf
# [...]
# BUILD SUCCEEDED - starting your binary
# 5 local devices
# [...]

image.bootable_disk(
    name = "demo",
    package_manager = "tine//catalog:fedora.rawhide.package-manager",
    definitions = image.DEFAULT_USR_VERITY_PARTITIONS,
    version = "0.0.0",
    package_sets = ["bootable"],
    packages = ["bash"],
    ops = [
        image.copy(":duf[duf]", "/usr/bin/duf"),
    ],
)

We can ask Buck2 for all available build targets in a cell.

tine/bin/tine buck targets //...
# root//:demo
# root//:demo.initrd
# root//:duf
# root//:duf.git
# root//:example.git
# root//:go.box
# root//:go.box.exec

image.vm(
    name = "demo-vm",
    autologin = "root",
    # re-using tine's rawhide box which has all of QEMU etc. installed
    box = "tine//catalog:fedora.rawhide.box",
    image = ":demo",
    credentials = {
        "firstboot.timezone": "UTC",
    }
)

This single command from a clean tree will then build the Go project, the image, and boot it.

tine/bin/tine buck run :demo-vm

You should see the following.

[  OK  ] Reached target graphical.target - Graphical Interface.
Fedora Linux 46 (Rawhide Prerelease)
Kernel 7.3.0-0.rc3.260916g9b87fdc9af2f.34.fc46.x86_64 on an x86_64 (hvc0)
fedora login: root (automatic login)
-bash-5.3# duf --help
Usage of duf:
      --all   include pseudo, duplicate, inaccessible file systems
[...]
-bash-5.3# ...
-bash-5.3# systemctl poweroff

SBOMs for Free

tine/bin/tine buck build --out /tmp/demo.cdx.json :demo[sbom][cyclonedx]

There’s Much More!

This blog post has only covered the basics. In order to get a fuller picture, please refer to the documentation. We think the following topics are the most helpful to get started.

The distro package machinery for importing, modifying, and syncing/merging packages from Fedora or Arch
Setting up a shared build cache, and the details of its threat model and design
Signing builds with hardware keys through PKCS#11
Including auditable cargo projects and Go projects into image builds
Using tine mount for rapid iteration on a built third-party git project component
tine comes with builtin tools for bumping dependencies and refreshing the catalog which you can re-use in your project to automate the regular housekeeping and get tested PRs for them, like this one
tine also supports Arch, BTW!

The next post in this series will be about the update and provisioning system we have built to distribute these and other images. We hope to see you then!

From Thin Air to Bootable Images: The tine Build System

Table of contents

Our Requirements