Continuous benchmarks for multi-device, copy-on-write filesystems — btrfs,\nZFS, bcachefs — measuring the things single-device ext4-style benchmarks\n(Phoronix et al.) never touch: redundancy layouts, snapshot aging and scaling,\ntransparent compression, encryption (native vs LUKS), reflinks, fsync tail\nlatency, degraded operation and rebuild, corruption self-healing, and\nnear-full/ENOSPC behavior — with ext4/xfs over md/LVM as the classic-stack\nbaselines.
\nClassic filesystem benchmarks run fio on one device with default mkfs options.\nThat says nothing about what modern filesystems are actually deployed for.\nThis suite benchmarks the machinery:
\n| Phase | What it measures |
|---|---|
| host calibration | fio on the runner's own disk before any filesystem exists — a VM-noise anchor |
| seq / rand write, rand read | baseline throughput on the chosen redundancy layout |
| trivial-op latency under load | "how long until my prompt comes back": a 4k write+fsync every 200ms (shell history, editor swap), p99 and worst case — idle, then while a 1M streaming writer floods the filesystem; CoW commit storms live here |
| source-tree ops | create / cold cp -r / rm -rf of a 20k-small-file tree — the "copy a kernel tree" test |
| large-directory scalability | create 100k empty files in one directory, enumerate names cold, stat every entry cold and warm, then delete — directory indexing and inode-cache behavior that tree-shaped workloads miss (LARGEDIR_FILES=1000000 reproduces the million-file variant) |
| parallel random read | same cold-cache read with 4 concurrent threads — a mirror can only serve from both copies under concurrency, so this is where replica read-scaling shows (on real hardware; CI loop devices share one disk and physically can't) |
| fsync tail latency | p99 / p99.9 fdatasync completion latency from the random-write phase — CoW transaction commits (ZFS txg, btrfs commit interval) spike periodically in ways the IOPS average hides |
| snapshot aging | random-overwrite bandwidth as snapshots accumulate (CoW fragmentation cost) — 100 snapshots where the technology allows; ZFS at 128K recordsize pins ~the whole file per snapshot so its default-recordsize layouts run 10, and old-style LVM snapshots amplify every origin write per snapshot so lvm layouts run 8 (both caps are findings, not shortcuts) |
| snapshot create | metadata cost of taking a snapshot |
| snapshot delete + reclaim | delete latency, foreground write bandwidth while background cleaning runs, time until the space actually returns |
| compression | zstd ratio + write throughput on 75%-compressible data |
| reflink | cp --reflink=always of a large file |
| clone divergence | the unshare penalty: the same 4k-overwrite workload into a plain file, a fresh reflink clone (btrfs/bcachefs/ZFS/xfs), and a freshly-snapshotted file (CoW filesystems and LVM) |
| degraded + rebuild | fail one device: IO while degraded, then time the rebuild onto a spare |
| snapshot-count scaling | 500 snapshots with no churn between them: create latency at the tail, snapshot-list time, remount time, bulk delete (native-snapshot filesystems) |
| near-full / ENOSPC | on a fresh small array of the same layout: write throughput near 95% and 99% full, then fill to hard ENOSPC — can you still delete (CoW needs free space to delete), and does deleting make the fs writable again? Caveat: btrfs hits its chunk-allocation wall before df crosses the target on small devices (1G data chunks are a big fraction of a CI-sized array — on multi-TB disks the same wall sits at 99.9%), so its probes run at the wall; the actual fullness at each probe is recorded in the JSON (nearfull*_pct) |
| corruption + scrub | write a 2G raw range onto one device behind the filesystem's back, scrub, then compare one tracked test-file hash; this probes checksum/redundancy recovery but is not a whole-filesystem health check, and the overwritten range can include allocated or unused space |
Results are published as a dashboard: https://bartosz.fenski.pl/modern-fs-benchmark/\n— per-metric charts sorted best-first, aging curves, and trends across runs,\nfilterable by filesystem family and layout class (e.g. "btrfs vs bcachefs,\nmulti-device only"), with linear/log scale switching and a sortable table.\nRun history lives on the results-data branch.
The suite uses the established fio tool and conventional MiB/s, IOPS, and\nlatency units, but the exact workload recipes and composite summary index are\nproject-specific rather than an industry-standard benchmark. It does not use\nO_DIRECT: documented cold-read phases drop caches before normal buffered\nreads, while other phases use normal buffered I/O with explicit durability\nbarriers where noted.
Every result records the exact tools and kernel-module versions tested —\nessential for ZFS and bcachefs, which are out-of-tree, where the kernel\nversion alone doesn't identify what actually ran. Shown in the dashboard\ntable, stored in the JSON.
\nDefault matrix — 26 configurations (4 devices, plus baselines; the\nauthoritative list is the matrix in .github/workflows/bench.yml):
ext4 on LVM raid10 — layered stack with block-layer CoW snapshots,
so the snapshot-aging phase is comparable with the native-CoW filesystems
xfs single / on md raid10 / on LVM raid10 — the same three stacks again;
XFS additionally has reflink, unlike ext4
btrfs / bcachefs / ZFS single-device — the CoW filesystems without
redundancy, head-to-head with ext4/xfs single: the pure cost (and features)\nof CoW itself. btrfs uses -m single: mkfs defaults to DUP metadata on a\nsingle device, which would double its metadata writes vs every other\nsingle-device row (community catch)
Encryption variants — ZFS native per-dataset AES-256-GCM
(mirror-enc), bcachefs native whole-fs ChaCha20/Poly1305\n(replicas2-enc), btrfs over one LUKS layer per device\n(raid1-luks — no native option, so every replica is encrypted\nseparately), and ext4 over a single LUKS layer on top of md\n(md-raid10-luks — the classic stack encrypts once, above the raid).\nCompression runs on all of them, so encrypt-after-compress vs\nopaque-blocks falls out of the existing zstd phase
-d raid1 -m raid1Single-parity — zfs raidz1, plus raidz1-enc with native encryption
on top (community request)
Dual-parity (raid6-class) — zfs raidz2 (and raidz2-enc,
community request), btrfs -d raid6 -m raid1c3\n(parity metadata is discouraged — write hole), ext4 on md raid6, and\nbcachefs --erasure_code --replicas=3 (stable since 1.37; write-hole-free\nby design — writes replicate first, background reconcile stripes them)\n(community request, incl. the correction that EC is no longer experimental)
xfs on a ZFS zvol — the Franken-stack people actually run: XFS
semantics on top; ZFS snapshots (fsfreeze-consistent), self-healing,\nand compression underneath (community request)
xfs on LVM raid10 + dm-integrity (--raidintegrity y) — per-sector
checksums give the classic stack detection AND correction: the fairest\nclassic-vs-CoW comparison in the corruption phase, with the performance\ntax quantified (community request)
ZFS — striped mirror pairs (raid10-like), at the default 128K recordsize
and again at recordsize=8k — one-variable proof of how much of ZFS's\nsmall-random-write cost is configuration, not design
bcachefs — --replicas=2 (kernel module built via DKMS from
apt.bcachefs.org since bcachefs left mainline in 6.17)
\nIf you want to try bcachefs as a working storage system rather than just\nbenchmark it, NASty is a NixOS-based\nNAS appliance built around it and a practical place to start.
Benchmark charts invite "which is fastest". For long-term storage that is\nthe wrong question — the right one is which stack tells you the truth\nabout your data, and it's why this suite exists (the corruption phase\nre-proves it every couple of hours):
\n**ext4/xfs on md or LVM raid — the default "safe" Linux setup — has no
data checksums.** Raid protects against a missing disk, not a lying\none: when a copy goes bad (disk firmware, cable, controller, bad RAM,\npower cut mid-write), the array cannot tell which copy is right. In our\ncorruption test these stacks have returned garbage to the application\nwith no error whatsoever — reads succeed, exit codes are 0, and the\ndamage can propagate into backups silently. A scrub counts mismatches;\nit cannot say which side is correct. An intact hash in one run is not proof\nof repair: read balancing may simply select the good copy.
btrfs, ZFS, and bcachefs verify every read against checksums and,
given appropriate redundancy, can reconstruct a bad allocated block from a\ngood replica or parity. Published redundant-layout runs have kept the tracked\ntest file readable and hash-identical, including native-encryption and parity\nlayouts. Nonzero found counts show detected damage, while nonzero repaired\ncounts provide the strongest evidence of reconstruction. Zero or unavailable\ncounts do not prove that the raw overwrite hit the tracked file.
The classic stack can buy the same guarantee — LVM raid with
--raidintegrity y (dm-integrity) is the first classic layout to pass\nour corruption phase — but almost nobody runs it, and the performance\ntax is measurable (that's the xfs/lvm-raid10-int row).
Speed matters and we measure it honestly. But if you keep data you care\nabout — photos, archives, the family's one copy of anything — on a\nnon-checksumming stack, no benchmark number compensates for corruption you\nwon't discover until years later. That risk is invisible in every classic\nfilesystem benchmark; here it's a first-class result\n(corruption + scrub on the dashboard).
\nThe dashboard's corruption badge is deliberately scoped. SURVIVED means the\nsingle tracked read.dat file was still readable and had the same MD5 after\nscrub; FAIL means it differed or could not be read; UNPROVEN means the hash\nmatched on a stack without end-to-end data checksums. The test does not hash the\nentire filesystem, prove every overwritten byte was allocated, or perform a\npost-corruption remount for every backend.
Every push/2-hourly cron builds each filesystem across 4 loop devices backed by\nsparse files, runs the suite, and publishes a results table in the job summary\nplus JSON artifacts. Each job's artifact also contains a full command trace\n(raw/<config>-trace.log) — every command executed, arguments fully expanded,\nwith source file and line — so "what exactly was run" is never a question.\n(BENCH_TRACE=1 mirrors it into the live log instead.)
Interpret CI numbers carefully. Runners are shared VMs and all "devices"\nlive on one virtual disk, so absolute MiB/s is meaningless and RAID striping\ngains are fiction. Matrix jobs also run in parallel, each on its own\nephemeral VM — so comparing filesystem A against filesystem B compares two\ndifferent machines. Mitigations, from strongest signal to weakest:
\nWithin-job ratios and shapes (aging curve slope, compression on/off,
degraded vs healthy) — same VM, same disk, directly meaningful.
Every job runs a host calibration first (fio on the runner's disk,
before any filesystem exists). Jobs on VMs below the calibration floor\n (CALIB_MIN_*, ~25% of runners' normal disk speed margin) fail fast\n and are automatically rerun on a fresh runner (up to 3 attempts) —\n junk numbers from an unlucky VM never enter the results.
Cross-filesystem deltas within one run — treat small differences (tens of
percent) as noise; large ones (2×+) are usually real.
Trends over repeated runs (2-hourly cron + every push) average the VM
lottery out — this is where cross-filesystem conclusions belong.
The same scripts take real block devices — this is where absolute numbers\nbecome valid:
\nsudo BENCH_DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde" BENCH_WIPE=1 \\\n scripts/run-bench.sh btrfs raid1Safety: devices must be unmounted, and anything carrying a filesystem\nsignature is refused unless BENCH_WIPE=1. Listed devices are wiped.
For an unmanaged hardware run, invoke scripts/run-bench.sh directly with\nBENCH_DEVICES, BENCH_SPARE_DEVICE, and BENCH_WIPE=1. The dedicated\nself-hosted GitHub workflow instead requires the NixOS module below; it never\naccepts device paths from workflow inputs. Workload sizes can be adjusted with\nSEQ_SIZE, AGING_SIZE, AGING_ITERS, and the other documented environment\nvariables. CI defaults are sized for 4×16 GB loop files.
This repository is also a flake with a reusable NixOS module and benchmark\npackage. Cluster configurations can import nixosModules.modern-fs-benchmark;\nthe module installs a dedicated Actions\nrunner, the filesystem tools and matching out-of-tree modules for the\ncluster-selected kernel, and a restricted root wrapper with fixed device paths.\nIt deliberately does not select a kernel or configure machine-wide boot,\nnetworking, users, or partitioning.
{\n inputs.modern-fs-benchmark.url =\n "github:fenio/modern-fs-benchmark";\n\n # In the target node's modules list:\n services.modern-fs-benchmark = {\n enable = true;\n repository = "https://github.com/fenio/modern-fs-benchmark";\n tokenFile = "/run/secrets/modern-fs-benchmark-runner";\n runnerName = "farm3";\n hardwareProfile = "farm3";\n runnerLabels = [ "fs-benchmark" ];\n devices = [\n "/dev/disk/by-partlabel/fsbench-nvme0-a"\n "/dev/disk/by-partlabel/fsbench-nvme1-a"\n "/dev/disk/by-partlabel/fsbench-nvme0-b"\n "/dev/disk/by-partlabel/fsbench-nvme1-b"\n ];\n spareDevice = "/dev/disk/by-partlabel/fsbench-nvme0-spare";\n zfsSingleDevice = "/dev/disk/by-partlabel/fsbench-nvme0-zfs-single";\n };\n}The four member devices and spare must each be exactly 16 GiB. The dedicated\nzfsSingleDevice must be exactly 32 GiB, matching the hosted-runner matrix.\nFor an unregistered manual run, the same immutable package is available as\nnix run .#manual -- <fs> <layout>; provide the documented BENCH_*\nenvironment variables and run it as root.\nSet BENCH_HARDWARE_RANDOM_SCALING=1 to include the optional 8- and 16-worker\nrandom read/write measurements and the 4/8/16-worker shard-aware write series\nthat the managed hardware wrapper enables.
The master cluster flake owns the node assignment and deploy-rs deployment, so\nthe runner can move to another machine without changing benchmark code. The\ndedicated bench-real-hw.yml workflow targets the fs-benchmark label and\nuses only the module's fixed devices. It publishes the farm3 history to\nresults-real-hw and the dashboard under /real-hw/; the existing bench.yml\nworkflow remains hosted-only and continues publishing results-data at the\nroot dashboard. Hardware runs can be dispatched manually. The weekly schedule\nis enabled only when the repository variable ENABLE_HARDWARE_BENCHMARKS is\nset to true. The token file should contain a fine-grained PAT because\nephemeral runners re-register after every job.
The rotational sas-hdd profile is routed independently through the\nfs-benchmark-sas-hdd runner label. Its workflow, enable variable, history,\nand dashboard are respectively bench-real-hw-sas-hdd.yml,\nENABLE_SAS_HDD_BENCHMARKS, results-real-hw-sas-hdd, and\n/sas-hdd/. Every result carries hardware_profile: "sas-hdd", and\npublication rejects a missing or mismatched profile so results from different\nmachines cannot enter the same trend series.
The separate hybrid-tier-v1 scenario compares\nBtrfs over mirrored dm-cache, ZFS special/L2ARC classes, and native bcachefs\nforeground/background/promote targets. It publishes independently under\n/sas-hdd/hybrid-tier/ and never expands the default hosted matrix.
The plan is bigger than loop devices. CI is the regression-tracking\nharness; the goal is to gather dedicated hardware and run the REAL tests\nthere — including the tiered topologies these filesystems were built for and\nthat no publication benchmarks today: NVMe cache/metadata in front of\nrotational data disks (bcachefs foreground/background targets, ZFS\nspecial/log/cache vdevs, LVM dm-cache with writeback and writethrough),\nmixed-rotational RAID, and how each setup behaves degraded and while\nrebuilding. Same suite, same JSON, same dashboard — only the device lists and\ntopology descriptions change.
\nRegular mixed-media runs depend on access to a machine whose benchmark devices\nmay be wiped. A suitable dedicated server currently costs approximately\n€70 per month, depending on availability and its exact disk configuration.\nIf recurring sponsorship covers that cost, the server specification,\nconfiguration, raw results, and command traces will all remain public.
\nI am also open to hardware support in other forms:
\nremote root access to an isolated Linux machine with clearly identified,
wipeable HDD, SSD, or NVMe devices;
If a suitable remote server does not become available, I may eventually build\nand host a runner myself as the budget allows. Financial support is available\nthrough GitHub Sponsors and\nKo-fi. To offer hardware or discuss a topology,\nopen an issue. Sponsors\nand hardware providers can be acknowledged if they wish, but do not receive\neditorial control over the methodology or results.
\nsudo scripts/install-deps.sh btrfs\nsudo scripts/run-bench.sh btrfs raid1\nscripts/summarize.sh results/result-*.jsonscripts/run-bench.sh orchestrates the phases, emits results/result-<fs>-<layout>.json\nscripts/lib/common.sh device layer (loop files or BENCH_DEVICES), LUKS helpers,\n corruption injection, fio helpers, default fs hooks\nscripts/lib/layered.sh shared md/LVM assembly, snapshots, degrade/repair (ext4 + xfs)\nscripts/fs/<fs>.sh per-filesystem backend\nscripts/install-deps.sh Debian/Ubuntu package setup per filesystem\nscripts/summarize.sh JSON results → markdown table (job summaries)\nscripts/make-dashboard.py results history → the static dashboard page\nscripts/audit-results.py anomaly scan over the results history — impossible\n orderings, self-healing failures, ENOSPC regressions,\n unexpected nulls (daily via the results-audit workflow)\nscripts/result-schema.json machine-readable result keys, types, capabilities, and display metadata\nscripts/result_schema.py shared result schema loading and validation\nscripts/validate-result.py validates result JSON against that contractResult documents carry schema_version; historical unversioned documents are\ntreated as version 1 so new metrics do not invalidate the stored history.
Adding a filesystem = one file in scripts/fs/ implementing fs_setup,\nfs_snapshot, and fs_teardown; everything else (fs_setup_compression,\nfs_compress_ratio, fs_snapshot_delete_all, fs_remount, fs_snap_list,\nfs_snapscale_delete, fs_degrade, fs_rebuild, fs_scrub, fs_version,\nfs_drop_caches, fs_free_bytes) has safe defaults in lib/common.sh and is\noptional — unimplemented hooks simply record null for their metrics.
This suite is deliberately open-ended — if you have opinions on what to\ntest and how, please open an issue or PR:
\nworkloads that would expose behavior the current phases miss
(databases, VM images, send/receive, metadata-heavy trees, …)
extra configurations and tuning you want measured: mount options,
recordsize/extent knobs, compression algorithms and levels, RAID\nprofiles, SLOG/special vdevs, nodatacow, …
fairness problems in the methodology — if a filesystem is being
measured in a way that misrepresents it, that's a bug here
additional filesystems or layered stacks (a backend is one small file
in scripts/fs/)
Tuned variants sit next to the defaults in the same matrix (see\nzfs mirror-8k), so every suggestion becomes a directly comparable row.
CoW-specific phases (the behaviors nothing mainstream benchmarks):
\nsend/receive: full + incremental stream throughput (btrfs, ZFS);
rsync over the classic stack as the contrast; bcachefs: not availablePartial device loss: device disappears briefly and returns — md
write-intent bitmaps, ZFS delta resilver, bcachefs journal catch-up;\na different (and common) recovery scenario than full-device rebuildlvm-thin layouts: our lvm rows use old-style snapshots, which are
the known-bad strawman — thin pools are what modern LVM users run, with\nproper CoW snapshots that should survive the aging and snapshot-scaling\nphases at full count. Next up."The Tower": ext4/xfs on lvm-thin on dm-vdo on LUKS on raid on
dm-integrity — the full feature-parity classic stack (checksums +\nredundancy + encryption + compression/dedup + CoW snapshots from five\ndm layers), versus the integrated filesystems that do it in one.\nRequested with a 😜 but taken seriously: dm-vdo is mainline since 6.9\nand thin-on-VDO is a documented configuration.Stratis (XFS on dm-thin/dm-integrity/dm-crypt, managed) — arguably
the closest classic-stack analogue to btrfs (community suggestion)NVMe cache tiers (community request): bcachefs
foreground/background targets, ZFS special/log/cache vdevs, LVM\ndm-cache — needs real mixed hardware; on CI loop devices both "tiers"\nare the same cloud SSD, so an honest cache benchmark is impossible\nthere (see *Real hardware* above)ext4 fscrypt variant (directory-level encryption — the third model
next to native and block-layer)Infrastructure:
\nKernel matrix: boot mainline kernels in qemu (runners support nested KVM)
and track behavioral regressions per kernel releasebtrfs/raid1-luks degraded phase (loop-detach can't fail a dm-crypt
mapper — needs the dm-error wrapper trick the lvm layouts use)Parse bcachefs scrub found/repaired counts (verdict via md5 works;
the counts aren't in 1.38-tools output)Copyright 2026 Bartosz Fenski.
\nSource code, configuration, workflows, and documentation are licensed under the\nApache License 2.0. Published benchmark result datasets, including\nthe results-data, results-real-hw, and results-real-hw-sas-hdd history\nbranches, are licensed under\nCreative Commons Attribution 4.0 International.