{"article":{"slug":"making-atuin-sync-32x-faster-with-packfiles","title":"Making Atuin sync 32x faster with packfiles","subtitle":null,"summary":"Atuin explains how client-built packfiles work around end-to-end encryption constraints to make Atuin Hub sync 32x faster (16x for self-hosted) starting with release 18.20.0.","content_type":"changelog","language":"en","canonical_url":"https://blog.atuin.sh/making-atuin-sync-32x-faster-with-packfiles/","author":{"name":"Marko Vejnovic","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Atuin","url":"https://atuin.sh/","listing_slug":null,"listing":null},"topics":[{"name":"developer-tools","slug":"developer-tools","url":"https://listedarticles.com/topics/developer-tools"},{"name":"shell","slug":"shell","url":"https://listedarticles.com/topics/shell"},{"name":"performance","slug":"performance","url":"https://listedarticles.com/topics/performance"},{"name":"encryption","slug":"encryption","url":"https://listedarticles.com/topics/encryption"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":1196,"reading_minutes":5,"published_at":"2026-08-25T21:05:59.000Z","added_at":"2026-11-16T10:54:04.041Z","updated_at":"2026-11-16T10:54:04.041Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/making-atuin-sync-32x-faster-with-packfiles","markdown_url":"https://listedarticles.com/articles/making-atuin-sync-32x-faster-with-packfiles.md","example":false,"citation":"Marko Vejnovic, Atuin. \"Making Atuin sync 32x faster with packfiles.\" 25 Aug 2026. https://blog.atuin.sh/making-atuin-sync-32x-faster-with-packfiles/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://blog.atuin.sh/making-atuin-sync-32x-faster-with-packfiles/"},"body_markdown":"# Making Atuin sync 32x faster with packfiles\n\nBy Marko Vejnovic · August 25, 2026\n\n> TL;DR: Atuin sync is now 32x faster. We sync many thousands of records in single-digit seconds.  \n>   \n> Get the latest Atuin with `curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh`\n\nAtuin syncs shell history and other terminal data end-to-end encrypted, meaning the server cannot read, compress, or bundle your data. The client now builds packfiles to work around this constraint. Along with other optimisations, we’re happy to share that Atuin Hub sync is now 32x faster, and for self-hosted users sync is 16x faster!\n\n![](https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-is-now-32-faster.png)\n\nWith the 18.20.0 release, sync has received many improvements. *Packfiles* collect multiple records and get stored in fast [Tigris](https://www.tigrisdata.com/?ref=blog.atuin.sh)-backed object storage. Just like regular history entries, packfiles are fully encrypted with the same [PASETO-based algorithm](https://blog.atuin.sh/new-encryption/) we use for records. Additionally, we’ve pipelined record downloads and we’ve also enabled batched SQL inserts. Best of all – *we managed to avoid updating the sync protocol version*.\n\n> This is not just a small performance improvement. This is a paradigm shift in how Atuin does sync.  \n>   \n> — lots of numbers on a GPU, August 2026\n\nThis blog post is going to go into some more technical details, and if you don’t care, look away!\n\n## Packfiles\n\nThe median user on [Atuin Hub](https://hub.atuin.sh/?ref=blog.atuin.sh) has 7,338 records. For that user, Atuin sync takes about 11.7s. The average user has 26,075 records, meaning their sync takes about 40s – tolerable, but not what we want. Atuin power users, however – the 95th and 99th percentile – experience sync times of about 2.2min and 5.4min, respectively. We set a higher bar:\n\n> *A sync of 1K records won’t take more than 3s and each additional 10K records won’t add more than 1s to that.*\n\n![Atuin 18.19.0 sync times. Chart demonstrates dissatisfying sync performance above 1000 records.](https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-18-19-0-sync-times--7-.png)\n\nWhat this is telling us is that Atuin’s sync doesn’t serve its most active users as well as it can – performance degrades as the record store becomes larger. Sync is inherently linear in the number of records, so we can’t change the scaling. But we can reduce the constant!\n\nAtuin’s record store is in many ways similar to a [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging?ref=blog.atuin.sh) – each new record is an addition to the log. A download sync means downloading each record the client is missing from the server. Atuin 18.19.0 downloads records in pages of 100. Each page contains an array of encrypted records. Unfortunately, for the most part, these pages are uncompressed, and what’s worse, ciphertext is indistinguishable from noise, so it does not compress.\n\nWe needed a more clever approach and took some inspiration from Git packfiles (though ours are simpler). The idea behind Atuin’s *packfiles* is the same: *take N plaintext records, bundle them, compress them and then encrypt them.*\n\nAn astute reader may have already noticed a problem – Atuin servers do not have access to the encryption key! Therefore, the *client* must be responsible for packing, as it is the only machine which has access to the records and the encryption key necessary to encrypt the packfile.\n\n#### Packing and uploading\n\nThe Atuin daemon now keeps track of a rolling buffer of loose records. As you use Atuin, this buffer will fill up. When it reaches a certain threshold, Atuin daemon will take the rolling buffer and create a new “packfile manifest” record, which encodes the window of records that the packfile contains. Note that no computationally heavy task has occurred here – we just kept track of records and created tiny manifest objects in the database for them.\n\nWhenever the daemon attempts to sync (or you manually invoke `atuin sync`), our new [capabilities system](https://github.com/atuinsh/atuin/pull/3797?ref=blog.atuin.sh) enables Atuin to check whether the server supports packfiles. If that’s the case, Atuin will perform packing – it will find each record the manifest describes, bundle them all together with [`rmp`](https://docs.rs/rmp?ref=blog.atuin.sh), compress with [`zstd`](https://docs.rs/zstd?ref=blog.atuin.sh) and finally encrypt with the existing PASETO v4 scheme.\n\nPackfiles are uploaded out-of-band with the regular records. Servers which support packfiles need to have special endpoints, so special logic and special handling is necessary.\n\n> At this time, **only Atuin Hub supports packfiles**. Self-hosted servers do not have a conforming implementation. We will bring these improvements to the self-hosted server once they stabilise.\n\nUploading packfiles means requesting the server provide a pre-signed URL the client can `PUT`.\n\n#### Downloading and sync\n\nThis is the fun part – when performing sync, we first sync packfiles. This means grabbing the packfile manifest records from the server, and then, for each manifest, querying the server for a pre-signed download URL. The client then downloads the packfile from the pre-signed URL, decrypts it, decompresses it, and then splits it up into individual records to store in its own table.\n\n#### Results\n\nWe ran synthetic benchmarks with different decades of record sizes. Between all the fixes, 18.20.0 improves performance by about 30 times for the 100K power-user. Packfiles constitute to about a 2 times improvement there.\n\n![Atuin 18.20.0 sync improvement chart, demonstrating a performance improvement of around 30 times for repositories of 100000 history entries.](https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-18-20-0-sync-speedup--6-.png)\n\nThese new sync protocols easily meet our thresholds.\n\n#### Future improvements\n\nWe are exploring the idea of “packfiles of packfiles” – a packfile can wrap multiple packfiles, instead of records alone. Currently, *Atuin Hub* packfiles are all 1000 records wide (and come out to be about 10–100 KB), but for use cases with very high numbers of packfiles, bundling older packfiles into trees of packfiles, enables further compression and latency reduction.\n\nPackfile downloads are pipelined, just as regular record downloads now are, however, a further tail-latency reduction can be achieved by requesting a single pre-signed URL valid for multiple packfiles, reducing the number of round-trips.\n\nFinally, we're exploring the idea of moving away from PASETOv4 towards plain [ChaCha20-Poly1305](https://en.wikipedia.org/wiki/ChaCha20-Poly1305?ref=blog.atuin.sh) which lets us encrypt and decrypt packfiles in streaming chunks and drop the PASETO base64 and metadata overhead.\n\nThe infrastructure we’ve built enables us to capture a lot more in your history than just the commands you ran. In a future release, Atuin will be able to capture, record and search through your commands’ *output, agent sessions* and more – made possible by the infrastructure necessary for packfiles.\n\n## Other improvements\n\nAs part of this release, we’ve done a huge amount of other work too.\n\n### Pipelining\n\nHistorically, Atuin would sequentially download a page of records, and then insert each record into the database. It would then grab the next page, repeat the inserts, and so on.\n\nA common strategy in improving such algorithms is [pipelining](https://en.wikipedia.org/wiki/Pipeline_(computing)?ref=blog.atuin.sh). [We had realized](https://github.com/atuinsh/atuin/pull/3955?ref=blog.atuin.sh) that if we made just a small adjustment in how we iterated during downloads, we could eagerly request pages from the server, in parallel.\n\n> Pipelining alone improved sync speed by about **340%**.\n\nThis feature is universally available in Atuin 18.20 – all you need to do is `atuin update`.\n\n## Conclusion\n\nWe’re really excited about this release – updating Atuin will immediately give you a sync that is 16x times faster. We’re rolling out packfiles incrementally over the next few weeks, so look forward to that!\n\nYou can update or install Atuin by running:\n\n```\ncurl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh\n```\n\nIf you would like to try packfiles today, you can enable them for your account [here](https://hub.atuin.sh/settings/feature-flags?ref=blog.atuin.sh).\n\nThanks!  \n— The Atuin team\n","body_html":"<h1 id=\"making-atuin-sync-32x-faster-with-packfiles\">Making Atuin sync 32x faster with packfiles</h1>\n<p>By Marko Vejnovic · August 25, 2026</p>\n<blockquote><p>TL;DR: Atuin sync is now 32x faster. We sync many thousands of records in single-digit seconds.  </p>\n<p>Get the latest Atuin with <code>curl --proto &#39;=https&#39; --tlsv1.2 -LsSf https://setup.atuin.sh | sh</code></p></blockquote>\n<p>Atuin syncs shell history and other terminal data end-to-end encrypted, meaning the server cannot read, compress, or bundle your data. The client now builds packfiles to work around this constraint. Along with other optimisations, we’re happy to share that Atuin Hub sync is now 32x faster, and for self-hosted users sync is 16x faster!</p>\n<figure><img src=\"https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-is-now-32-faster.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>With the 18.20.0 release, sync has received many improvements. <em>Packfiles</em> collect multiple records and get stored in fast <a href=\"https://www.tigrisdata.com/?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">Tigris</a>-backed object storage. Just like regular history entries, packfiles are fully encrypted with the same <a href=\"https://blog.atuin.sh/new-encryption/\" rel=\"nofollow ugc noopener\">PASETO-based algorithm</a> we use for records. Additionally, we’ve pipelined record downloads and we’ve also enabled batched SQL inserts. Best of all – <em>we managed to avoid updating the sync protocol version</em>.</p>\n<blockquote><p>This is not just a small performance improvement. This is a paradigm shift in how Atuin does sync.  </p>\n<p>— lots of numbers on a GPU, August 2026</p></blockquote>\n<p>This blog post is going to go into some more technical details, and if you don’t care, look away!</p>\n<h2 id=\"packfiles\">Packfiles</h2>\n<p>The median user on <a href=\"https://hub.atuin.sh/?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">Atuin Hub</a> has 7,338 records. For that user, Atuin sync takes about 11.7s. The average user has 26,075 records, meaning their sync takes about 40s – tolerable, but not what we want. Atuin power users, however – the 95th and 99th percentile – experience sync times of about 2.2min and 5.4min, respectively. We set a higher bar:</p>\n<blockquote><p><em>A sync of 1K records won’t take more than 3s and each additional 10K records won’t add more than 1s to that.</em></p></blockquote>\n<figure><img src=\"https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-18-19-0-sync-times--7-.png\" alt=\"Atuin 18.19.0 sync times. Chart demonstrates dissatisfying sync performance above 1000 records.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>What this is telling us is that Atuin’s sync doesn’t serve its most active users as well as it can – performance degrades as the record store becomes larger. Sync is inherently linear in the number of records, so we can’t change the scaling. But we can reduce the constant!</p>\n<p>Atuin’s record store is in many ways similar to a <a href=\"https://en.wikipedia.org/wiki/Write-ahead_logging?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">write-ahead log</a> – each new record is an addition to the log. A download sync means downloading each record the client is missing from the server. Atuin 18.19.0 downloads records in pages of 100. Each page contains an array of encrypted records. Unfortunately, for the most part, these pages are uncompressed, and what’s worse, ciphertext is indistinguishable from noise, so it does not compress.</p>\n<p>We needed a more clever approach and took some inspiration from Git packfiles (though ours are simpler). The idea behind Atuin’s <em>packfiles</em> is the same: <em>take N plaintext records, bundle them, compress them and then encrypt them.</em></p>\n<p>An astute reader may have already noticed a problem – Atuin servers do not have access to the encryption key! Therefore, the <em>client</em> must be responsible for packing, as it is the only machine which has access to the records and the encryption key necessary to encrypt the packfile.</p>\n<h4 id=\"packing-and-uploading\">Packing and uploading</h4>\n<p>The Atuin daemon now keeps track of a rolling buffer of loose records. As you use Atuin, this buffer will fill up. When it reaches a certain threshold, Atuin daemon will take the rolling buffer and create a new “packfile manifest” record, which encodes the window of records that the packfile contains. Note that no computationally heavy task has occurred here – we just kept track of records and created tiny manifest objects in the database for them.</p>\n<p>Whenever the daemon attempts to sync (or you manually invoke <code>atuin sync</code>), our new <a href=\"https://github.com/atuinsh/atuin/pull/3797?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">capabilities system</a> enables Atuin to check whether the server supports packfiles. If that’s the case, Atuin will perform packing – it will find each record the manifest describes, bundle them all together with <a href=\"https://docs.rs/rmp?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\"><code>rmp</code></a>, compress with <a href=\"https://docs.rs/zstd?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\"><code>zstd</code></a> and finally encrypt with the existing PASETO v4 scheme.</p>\n<p>Packfiles are uploaded out-of-band with the regular records. Servers which support packfiles need to have special endpoints, so special logic and special handling is necessary.</p>\n<blockquote><p>At this time, <strong>only Atuin Hub supports packfiles</strong>. Self-hosted servers do not have a conforming implementation. We will bring these improvements to the self-hosted server once they stabilise.</p></blockquote>\n<p>Uploading packfiles means requesting the server provide a pre-signed URL the client can <code>PUT</code>.</p>\n<h4 id=\"downloading-and-sync\">Downloading and sync</h4>\n<p>This is the fun part – when performing sync, we first sync packfiles. This means grabbing the packfile manifest records from the server, and then, for each manifest, querying the server for a pre-signed download URL. The client then downloads the packfile from the pre-signed URL, decrypts it, decompresses it, and then splits it up into individual records to store in its own table.</p>\n<h4 id=\"results\">Results</h4>\n<p>We ran synthetic benchmarks with different decades of record sizes. Between all the fixes, 18.20.0 improves performance by about 30 times for the 100K power-user. Packfiles constitute to about a 2 times improvement there.</p>\n<figure><img src=\"https://storage.ghost.io/c/66/46/6646b777-02d8-44b3-9d01-dd721730b905/content/images/2026/08/atuin-18-20-0-sync-speedup--6-.png\" alt=\"Atuin 18.20.0 sync improvement chart, demonstrating a performance improvement of around 30 times for repositories of 100000 history entries.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>These new sync protocols easily meet our thresholds.</p>\n<h4 id=\"future-improvements\">Future improvements</h4>\n<p>We are exploring the idea of “packfiles of packfiles” – a packfile can wrap multiple packfiles, instead of records alone. Currently, <em>Atuin Hub</em> packfiles are all 1000 records wide (and come out to be about 10–100 KB), but for use cases with very high numbers of packfiles, bundling older packfiles into trees of packfiles, enables further compression and latency reduction.</p>\n<p>Packfile downloads are pipelined, just as regular record downloads now are, however, a further tail-latency reduction can be achieved by requesting a single pre-signed URL valid for multiple packfiles, reducing the number of round-trips.</p>\n<p>Finally, we&#39;re exploring the idea of moving away from PASETOv4 towards plain <a href=\"https://en.wikipedia.org/wiki/ChaCha20-Poly1305?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">ChaCha20-Poly1305</a> which lets us encrypt and decrypt packfiles in streaming chunks and drop the PASETO base64 and metadata overhead.</p>\n<p>The infrastructure we’ve built enables us to capture a lot more in your history than just the commands you ran. In a future release, Atuin will be able to capture, record and search through your commands’ <em>output, agent sessions</em> and more – made possible by the infrastructure necessary for packfiles.</p>\n<h2 id=\"other-improvements\">Other improvements</h2>\n<p>As part of this release, we’ve done a huge amount of other work too.</p>\n<h3 id=\"pipelining\">Pipelining</h3>\n<p>Historically, Atuin would sequentially download a page of records, and then insert each record into the database. It would then grab the next page, repeat the inserts, and so on.</p>\n<p>A common strategy in improving such algorithms is <a href=\"https://en.wikipedia.org/wiki/Pipeline_(computing)?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">pipelining</a>. <a href=\"https://github.com/atuinsh/atuin/pull/3955?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">We had realized</a> that if we made just a small adjustment in how we iterated during downloads, we could eagerly request pages from the server, in parallel.</p>\n<blockquote><p>Pipelining alone improved sync speed by about <strong>340%</strong>.</p></blockquote>\n<p>This feature is universally available in Atuin 18.20 – all you need to do is <code>atuin update</code>.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>We’re really excited about this release – updating Atuin will immediately give you a sync that is 16x times faster. We’re rolling out packfiles incrementally over the next few weeks, so look forward to that!</p>\n<p>You can update or install Atuin by running:</p>\n<pre><code>curl --proto &#39;=https&#39; --tlsv1.2 -LsSf https://setup.atuin.sh | sh</code></pre>\n<p>If you would like to try packfiles today, you can enable them for your account <a href=\"https://hub.atuin.sh/settings/feature-flags?ref=blog.atuin.sh\" rel=\"nofollow ugc noopener\">here</a>.</p>\n<p>Thanks!<br />\n— The Atuin team</p>","headings":[{"level":1,"text":"Making Atuin sync 32x faster with packfiles","id":"making-atuin-sync-32x-faster-with-packfiles"},{"level":2,"text":"Packfiles","id":"packfiles"},{"level":2,"text":"Other improvements","id":"other-improvements"},{"level":3,"text":"Pipelining","id":"pipelining"},{"level":2,"text":"Conclusion","id":"conclusion"}]}}