By Daan De Meyer and Martin Pitt
\nThis post is part of a series covering some of the open source work we have been doing in recent months. Today we introduce and publish tine, our new Buck2-based build system.
\nBuilding an operating system with cryptographically verifiable integrity has to start with a build system with these very properties. At the same time, we are part of the greater open source community and want both to contribute and re-use as much existing work as possible. We also aim for efficient development with fast turnaround.
\nThis roughly translates into the following requirements for our build system:
\nBefore building our own build tool, we evaluated several options.
\nGiven our team includes the creator and maintainer of mkosi, it was a natural first candidate to evaluate. But we quickly came to the conclusion that it has some fundamental shortcomings. It’s great at building individual images based on upstream packages, but becomes restrictive when building several weakly related images or if you need more control over the artifacts that make up an image. Building multiple images is limited to images that are intended to be shipped as part of the “main” image.
We need to build many different kinds of artifacts in a uniform and robust way, not just images. Hence the build needs to be orchestrated by a generic and flexible tool. The main build file should be a language calling into library functions like “compile a cargo crate” or “build a UKI”. mkosi is the opposite, it’s a framework: It knows how to build images, and only gives you free-form opaque hooks for the other kinds of builds. That leads to a bad experience when you want to build more than just images.
\nThe Open Build Service (OBS) is a powerful fully-integrated build system that is primarily used by SUSE and the openSUSE project to produce all of their artifacts, anything from packages to ISOs, and many other image formats. It has strong dependency tracking and supports a dizzying array of distributions.
\nHowever, it’s also the antithesis of “minimal host requirements”. The server side of it is required, central, and non-trivial to self-host. It’s also not a generic build system, meaning any new artifact types would either have to be modelled as packages or require heavy patches to OBS. We concluded that this lack of flexibility combined with its overall architecture would make it difficult for OBS to meet our requirements.
\nApache BuildStream describes the operating system image as a graph of YAML "elements", each with its own sources, dependencies and build commands. BuildStream builds each one in a bubblewrap sandbox and caches the result under a hash of everything that went into it, similar to Buck2. It is mature and used to build freedesktop-sdk, GNOME OS, and WebKitGTK.
\nOur concerns with BuildStream are mostly around bootstrapping and extensibility. BuildStream is a Python application with compiled extensions and other dependencies. It relies on a separate set of helper programs, plus sandboxing tools from the host. Each of those can be pinned, but through different mechanisms, and even then the result still depends on the host's Python. In practice you run it from a pinned container image instead, but then you’re still dependent on an entire container runtime you don’t control.
\nBuildStream’s YAML is a plain data format, and is extended with Python plugins. YAML has no functions, so over time you end up copy-pasting across the project. Ultimately we decided to go for a tool with a better bootstrapping and pinning story as well as a more flexible language.
\nAntlir is Meta's OS image builder, built on top of the Buck2 build system engine. Buck2 is Meta's open source build system with emphasis on correctness, flexibility, and caching as much as possible. Antlir implements various rules for building images with Buck2.
\nAs Antlir is a high-level tool focused on Meta’s internal repository, it is naturally very opinionated and designed for Meta’s internal use cases. For example, it requires btrfs and is strongly focused on a single monorepo.
\nWhile we decided against using Antlir itself, its underlying engine, Buck2, turned out to be a good fit, and we ended up choosing it as the foundation for our own build system.
\nIn essence, tine is a set of opinionated Buck2 rules to build rpm, Rust crate and Go module components, UKIs, and images; it can sign images either with a hardware key through PKCS#11 or a locally generated key. The intent is to combine the best ideas from Antlir and mkosi into a single tool.
\ntine has only three requirements on its build host: git, python3 (just for its own bootstrapping, not for production builds) and user namespaces. From there, it bootstraps everything it needs from pinned declarations to get a reproducible and independent build environment. That can be a distribution as old or modern as you need.
\nAn important tine concept is the “box”, which is a declared and pinned down environment to run a build task. Think containers or distrobox, but declared natively in Buck2’s language, and using Buck2’s caching and rebuild rules, so they build quickly and naturally, stay reproducible, and need no further dependencies to run. tine itself defines boxes for running rpmbuild, go, or cargo, or a bigger multi-purpose one called fedora.rawhide.box which contains e.g. systemd-ukify for building images and QEMU for running virtual machines. Your own project can define its own boxes.
\nTo better understand this post and the examples, here is a one-minute Buck2 primer for those familiar with Make or Meson:
\nsubdir:name , or even just:name for a target in the current directory. A single target can publish several output variants (“subtargets”), e.g.:my_cool_os[qcow2] or:my_cool_os[sbom] .*_target() , or the structure of a Makefile rule: a Starlark expression which translates a target into a set ofactions and their parameters. It does not run anything by itself. For example, abootable_disk(name = “myos”, param1 = …) rule defines amyos target and invokes abootable_disk rule which translates it into actions like “install rpms”, “runsystemd-repart” and so on.More information can be found on Buck2’s key concepts page.
\nUnlike Make or Meson, Buck2 never decides what to rebuild from timestamps. An action is keyed by a hash of all of its inputs: the sources, the tool binaries, the build platform configuration, and the command line itself. That is what makes its incremental builds correct and trustworthy, and it also allows taking an action's result from a shared cache instead of re-running it.
\nLet’s walk through how to use tine in your own projects. We will build a very basic example from scratch: a bootable image based on Fedora Rawhide with a Go project, and boot it. In this example, we’ll use duf, a CLI tool that shows free/used disk space in a text terminal with nice ASCII art.
\nLet’s follow tine's README and set up a fresh demo git repository which pulls in tine and initializes it.
\ngit init tine-demo\ncd tine-demo\ngit submodule add https://github.com/amutable-systems/tine tine\ntine/bin/tine init\ngit add .\ngit commit -m "initialize"Now let’s add a BUCK file. We’ll walk through it in several blocks, but these all go in the same file. First we need to import some definitions. This is Starlark, so akin to Python’s import statements.
\nload("@tine//box:defs.bzl", "box")\nload("@tine//git:defs.bzl", "git")\nload("@tine//go:defs.bzl", "go")\nload("@tine//image:defs.bzl", "image")Next we need to define a build environment for the Go compiler. tine already offers a Fedora rawhide catalog. So, let’s just use that (hence the tine// cell) and Fedora’s golang package. A real project would likely define and track their parent OS catalog by itself, instead of blindly following tine’s.
\nbox.new(\n name = "go.box",\n packages = ["golang"],\n release = "tine//catalog:fedora.rawhide.release",\n)Declare the duf Go project git repository which we want to build. tine requires pinning every input exactly, so we specify a git commit ID. That git repository is then passed as input to the go.package() rule which binds the above go.box and the git checkout, both referenced as relative targets (see above), hence the colon separator.
\ngit.fetch(\n name = "duf.git",\n repo = "https://github.com/muesli/duf",\n rev = "4636deb4a7b707a9f04c602db033f9837e50b3f6",\n)\ngo.package(\n name = "duf",\n box = ":go.box", # a target in the current directory\n src = ":duf.git", # another target\n)With that we can already build and execute the binary.
\ntine/bin/tine buck run :duf\n# [...]\n# BUILD SUCCEEDED - starting your binary\n# 5 local devices\n# [...]And now for the last big piece: the bootable image. Just as with the Go box, we re-use the tine catalog’s package manager that gets packages from Fedora Rawhide. This is the minimum set to be able to boot in a virtual machine, plus bash. As an extra ops (operation) this installs the built hello binary from the above go rule.
\nimage.bootable_disk(\n name = "demo",\n package_manager = "tine//catalog:fedora.rawhide.package-manager",\n definitions = image.DEFAULT_USR_VERITY_PARTITIONS,\n version = "0.0.0",\n package_sets = ["bootable"],\n packages = ["bash"],\n ops = [\n image.copy(":duf[duf]", "/usr/bin/duf"),\n ],\n)We can ask Buck2 for all available build targets in a cell.
\ntine/bin/tine buck targets //...\n# root//:demo\n# root//:demo.initrd\n# root//:duf\n# root//:duf.git\n# root//:example.git\n# root//:go.box\n# root//:go.box.execAfter all of that, we can now build the image. However, it’s far more interesting to actually see it live in QEMU. Let’s add a VM definition, with auto-login for convenience, that boots our shiny new demo image using QEMU and related tools from tine’s own Rawhide catalog.
\nimage.vm(\n name = "demo-vm",\n autologin = "root",\n # re-using tine's rawhide box which has all of QEMU etc. installed\n box = "tine//catalog:fedora.rawhide.box",\n image = ":demo",\n credentials = {\n "firstboot.timezone": "UTC",\n }\n)This single command from a clean tree will then build the Go project, the image, and boot it.
\ntine/bin/tine buck run :demo-vmYou should see the following.
\n[ OK ] Reached target graphical.target - Graphical Interface.\nFedora Linux 46 (Rawhide Prerelease)\nKernel 7.3.0-0.rc3.260916g9b87fdc9af2f.34.fc46.x86_64 on an x86_64 (hvc0)\nfedora login: root (automatic login)\n-bash-5.3# duf --help\nUsage of duf:\n --all include pseudo, duplicate, inaccessible file systems\n[...]\n-bash-5.3# ...\n-bash-5.3# systemctl poweroffLook at tine’s examples/ directory for BUCK and auxiliary files for various scenarios such as a SecureBoot/verity OS signed by either a generated or hardware key, how to build Go/Rust projects, the various kinds of rules and customizations which tine offers, or how to write integration tests.
\ntine gives you a lot more for free. For example, it integrates the Syft SBOM generator so you can ask it to build a CycloneDX standard SBOM. Let’s also specify an output path, so that you don’t have to fish it out of buck-out/:
\ntine/bin/tine buck build --out /tmp/demo.cdx.json :demo[sbom][cyclonedx]This blog post has only covered the basics. In order to get a fuller picture, please refer to the documentation. We think the following topics are the most helpful to get started.
\nThe next post in this series will be about the update and provisioning system we have built to distribute these and other images. We hope to see you then!
\nThis post is part of a series covering some of the open source work we have been doing in recent months. Today we introduce and publish tine, our new Buck2-based build system.
\nBuilding an operating system with cryptographically verifiable integrity has to start with a build system with these very properties. At the same time, we are part of the greater open source community and want both to contribute and re-use as much existing work as possible. We also aim for efficient development with fast turnaround.
\nThis roughly translates into the following requirements for our build system:
\nBefore building our own build tool, we evaluated several options.
\nGiven our team includes the creator and maintainer of mkosi, it was a natural first candidate to evaluate. But we quickly came to the conclusion that it has some fundamental shortcomings. It’s great at building individual images based on upstream packages, but becomes restrictive when building several weakly related images or if you need more control over the artifacts that make up an image. Building multiple images is limited to images that are intended to be shipped as part of the “main” image.
We need to build many different kinds of artifacts in a uniform and robust way, not just images. Hence the build needs to be orchestrated by a generic and flexible tool. The main build file should be a language calling into library functions like “compile a cargo crate” or “build a UKI”. mkosi is the opposite, it’s a framework: It knows how to build images, and only gives you free-form opaque hooks for the other kinds of builds. That leads to a bad experience when you want to build more than just images.
\nThe Open Build Service (OBS) is a powerful fully-integrated build system that is primarily used by SUSE and the openSUSE project to produce all of their artifacts, anything from packages to ISOs, and many other image formats. It has strong dependency tracking and supports a dizzying array of distributions.
\nHowever, it’s also the antithesis of “minimal host requirements”. The server side of it is required, central, and non-trivial to self-host. It’s also not a generic build system, meaning any new artifact types would either have to be modelled as packages or require heavy patches to OBS. We concluded that this lack of flexibility combined with its overall architecture would make it difficult for OBS to meet our requirements.
\nApache BuildStream describes the operating system image as a graph of YAML "elements", each with its own sources, dependencies and build commands. BuildStream builds each one in a bubblewrap sandbox and caches the result under a hash of everything that went into it, similar to Buck2. It is mature and used to build freedesktop-sdk, GNOME OS, and WebKitGTK.
\nOur concerns with BuildStream are mostly around bootstrapping and extensibility. BuildStream is a Python application with compiled extensions and other dependencies. It relies on a separate set of helper programs, plus sandboxing tools from the host. Each of those can be pinned, but through different mechanisms, and even then the result still depends on the host's Python. In practice you run it from a pinned container image instead, but then you’re still dependent on an entire container runtime you don’t control.
\nBuildStream’s YAML is a plain data format, and is extended with Python plugins. YAML has no functions, so over time you end up copy-pasting across the project. Ultimately we decided to go for a tool with a better bootstrapping and pinning story as well as a more flexible language.
\nAntlir is Meta's OS image builder, built on top of the Buck2 build system engine. Buck2 is Meta's open source build system with emphasis on correctness, flexibility, and caching as much as possible. Antlir implements various rules for building images with Buck2.
\nAs Antlir is a high-level tool focused on Meta’s internal repository, it is naturally very opinionated and designed for Meta’s internal use cases. For example, it requires btrfs and is strongly focused on a single monorepo.
\nWhile we decided against using Antlir itself, its underlying engine, Buck2, turned out to be a good fit, and we ended up choosing it as the foundation for our own build system.
\nIn essence, tine is a set of opinionated Buck2 rules to build rpm, Rust crate and Go module components, UKIs, and images; it can sign images either with a hardware key through PKCS#11 or a locally generated key. The intent is to combine the best ideas from Antlir and mkosi into a single tool.
\ntine has only three requirements on its build host: git, python3 (just for its own bootstrapping, not for production builds) and user namespaces. From there, it bootstraps everything it needs from pinned declarations to get a reproducible and independent build environment. That can be a distribution as old or modern as you need.
\nAn important tine concept is the “box”, which is a declared and pinned down environment to run a build task. Think containers or distrobox, but declared natively in Buck2’s language, and using Buck2’s caching and rebuild rules, so they build quickly and naturally, stay reproducible, and need no further dependencies to run. tine itself defines boxes for running rpmbuild, go, or cargo, or a bigger multi-purpose one called fedora.rawhide.box which contains e.g. systemd-ukify for building images and QEMU for running virtual machines. Your own project can define its own boxes.
\nTo better understand this post and the examples, here is a one-minute Buck2 primer for those familiar with Make or Meson:
\nsubdir:name , or even just:name for a target in the current directory. A single target can publish several output variants (“subtargets”), e.g.:my_cool_os[qcow2] or:my_cool_os[sbom] .*_target() , or the structure of a Makefile rule: a Starlark expression which translates a target into a set ofactions and their parameters. It does not run anything by itself. For example, abootable_disk(name = “myos”, param1 = …) rule defines amyos target and invokes abootable_disk rule which translates it into actions like “install rpms”, “runsystemd-repart” and so on.More information can be found on Buck2’s key concepts page.
\nUnlike Make or Meson, Buck2 never decides what to rebuild from timestamps. An action is keyed by a hash of all of its inputs: the sources, the tool binaries, the build platform configuration, and the command line itself. That is what makes its incremental builds correct and trustworthy, and it also allows taking an action's result from a shared cache instead of re-running it.
\nLet’s walk through how to use tine in your own projects. We will build a very basic example from scratch: a bootable image based on Fedora Rawhide with a Go project, and boot it. In this example, we’ll use duf, a CLI tool that shows free/used disk space in a text terminal with nice ASCII art.
\nLet’s follow tine's README and set up a fresh demo git repository which pulls in tine and initializes it.
\ngit init tine-demo\ncd tine-demo\ngit submodule add https://github.com/amutable-systems/tine tine\ntine/bin/tine init\ngit add .\ngit commit -m "initialize"Now let’s add a BUCK file. We’ll walk through it in several blocks, but these all go in the same file. First we need to import some definitions. This is Starlark, so akin to Python’s import statements.
\nload("@tine//box:defs.bzl", "box")\nload("@tine//git:defs.bzl", "git")\nload("@tine//go:defs.bzl", "go")\nload("@tine//image:defs.bzl", "image")Next we need to define a build environment for the Go compiler. tine already offers a Fedora rawhide catalog. So, let’s just use that (hence the tine// cell) and Fedora’s golang package. A real project would likely define and track their parent OS catalog by itself, instead of blindly following tine’s.
\nbox.new(\n name = "go.box",\n packages = ["golang"],\n release = "tine//catalog:fedora.rawhide.release",\n)Declare the duf Go project git repository which we want to build. tine requires pinning every input exactly, so we specify a git commit ID. That git repository is then passed as input to the go.package() rule which binds the above go.box and the git checkout, both referenced as relative targets (see above), hence the colon separator.
\ngit.fetch(\n name = "duf.git",\n repo = "https://github.com/muesli/duf",\n rev = "4636deb4a7b707a9f04c602db033f9837e50b3f6",\n)\ngo.package(\n name = "duf",\n box = ":go.box", # a target in the current directory\n src = ":duf.git", # another target\n)With that we can already build and execute the binary.
\ntine/bin/tine buck run :duf\n# [...]\n# BUILD SUCCEEDED - starting your binary\n# 5 local devices\n# [...]And now for the last big piece: the bootable image. Just as with the Go box, we re-use the tine catalog’s package manager that gets packages from Fedora Rawhide. This is the minimum set to be able to boot in a virtual machine, plus bash. As an extra ops (operation) this installs the built hello binary from the above go rule.
\nimage.bootable_disk(\n name = "demo",\n package_manager = "tine//catalog:fedora.rawhide.package-manager",\n definitions = image.DEFAULT_USR_VERITY_PARTITIONS,\n version = "0.0.0",\n package_sets = ["bootable"],\n packages = ["bash"],\n ops = [\n image.copy(":duf[duf]", "/usr/bin/duf"),\n ],\n)We can ask Buck2 for all available build targets in a cell.
\ntine/bin/tine buck targets //...\n# root//:demo\n# root//:demo.initrd\n# root//:duf\n# root//:duf.git\n# root//:example.git\n# root//:go.box\n# root//:go.box.execAfter all of that, we can now build the image. However, it’s far more interesting to actually see it live in QEMU. Let’s add a VM definition, with auto-login for convenience, that boots our shiny new demo image using QEMU and related tools from tine’s own Rawhide catalog.
\nimage.vm(\n name = "demo-vm",\n autologin = "root",\n # re-using tine's rawhide box which has all of QEMU etc. installed\n box = "tine//catalog:fedora.rawhide.box",\n image = ":demo",\n credentials = {\n "firstboot.timezone": "UTC",\n }\n)This single command from a clean tree will then build the Go project, the image, and boot it.
\ntine/bin/tine buck run :demo-vmYou should see the following.
\n[ OK ] Reached target graphical.target - Graphical Interface.\nFedora Linux 46 (Rawhide Prerelease)\nKernel 7.3.0-0.rc3.260916g9b87fdc9af2f.34.fc46.x86_64 on an x86_64 (hvc0)\nfedora login: root (automatic login)\n-bash-5.3# duf --help\nUsage of duf:\n --all include pseudo, duplicate, inaccessible file systems\n[...]\n-bash-5.3# ...\n-bash-5.3# systemctl poweroffLook at tine’s examples/ directory for BUCK and auxiliary files for various scenarios such as a SecureBoot/verity OS signed by either a generated or hardware key, how to build Go/Rust projects, the various kinds of rules and customizations which tine offers, or how to write integration tests.
\ntine gives you a lot more for free. For example, it integrates the Syft SBOM generator so you can ask it to build a CycloneDX standard SBOM. Let’s also specify an output path, so that you don’t have to fish it out of buck-out/:
\ntine/bin/tine buck build --out /tmp/demo.cdx.json :demo[sbom][cyclonedx]This blog post has only covered the basics. In order to get a fuller picture, please refer to the documentation. We think the following topics are the most helpful to get started.
\nThe next post in this series will be about the update and provisioning system we have built to distribute these and other images. We hope to see you then!
","headings":[{"level":1,"text":"From Thin Air to Bootable Images: The tine Build System","id":"from-thin-air-to-bootable-images-the-tine-build-system"},{"level":2,"text":"Table of contents","id":"table-of-contents"},{"level":2,"text":"Our Requirements","id":"our-requirements"},{"level":2,"text":"Existing Tools","id":"existing-tools"},{"level":3,"text":"mkosi","id":"mkosi"},{"level":3,"text":"Open Build Service","id":"open-build-service"},{"level":3,"text":"BuildStream","id":"buildstream"},{"level":3,"text":"Antlir","id":"antlir"},{"level":2,"text":"Our Build System: tine","id":"our-build-system-tine"},{"level":3,"text":"tine’s Engine: Buck2","id":"tine-s-engine-buck2"},{"level":2,"text":"Walkthrough: Building a Bootable Image","id":"walkthrough-building-a-bootable-image"},{"level":2,"text":"SBOMs for Free","id":"sboms-for-free"},{"level":2,"text":"There’s Much More!","id":"there-s-much-more"},{"level":2,"text":"Our Requirements","id":"our-requirements-2"},{"level":2,"text":"Existing Tools","id":"existing-tools-2"},{"level":3,"text":"mkosi","id":"mkosi-2"},{"level":3,"text":"Open Build Service","id":"open-build-service-2"},{"level":3,"text":"BuildStream","id":"buildstream-2"},{"level":3,"text":"Antlir","id":"antlir-2"},{"level":2,"text":"Our Build System: tine","id":"our-build-system-tine-2"},{"level":3,"text":"tine’s Engine: Buck2","id":"tine-s-engine-buck2-2"},{"level":2,"text":"Walkthrough: Building a Bootable Image","id":"walkthrough-building-a-bootable-image-2"},{"level":2,"text":"SBOMs for Free","id":"sboms-for-free-2"},{"level":2,"text":"There’s Much More!","id":"there-s-much-more-2"}]}}