{"article":{"slug":"tar-codec-a-fast-and-secure-tar-parser-for-the-rust-ecosystem","title":"tar-codec: a fast and secure tar parser for the Rust ecosystem","subtitle":null,"summary":"Astral's William Woodruff explains tar-codec, the new Rust tar parser uv uses for source distributions from 0.13: roughly twice as fast as astral-tokio-tar, built to reduce parser differentials and other security risks, with a long tour of the messy UStar, pax and GNU tar formats, its state-machine design, safe extraction and lessons learned.","content_type":"blog_post","language":"en","canonical_url":"https://astral.sh/blog/tar-codec","author":{"name":"William Woodruff","url":"https://github.com/woodruffw","person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Astral","url":"https://astral.sh/","listing_slug":null,"listing":null},"topics":[{"name":"Rust","slug":"rust","url":"https://listedarticles.com/topics/rust"},{"name":"Python","slug":"python","url":"https://listedarticles.com/topics/python"},{"name":"Security","slug":"security","url":"https://listedarticles.com/topics/security"},{"name":"Open Source","slug":"open-source","url":"https://listedarticles.com/topics/open-source"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":5898,"reading_minutes":26,"published_at":"2026-10-27T00:00:00.000Z","added_at":"2026-10-27T23:14:26.553Z","updated_at":"2026-10-27T23:14:26.553Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/tar-codec-a-fast-and-secure-tar-parser-for-the-rust-ecosystem","markdown_url":"https://listedarticles.com/articles/tar-codec-a-fast-and-secure-tar-parser-for-the-rust-ecosystem.md","example":false,"citation":"William Woodruff, Astral. \"tar-codec: a fast and secure tar parser for the Rust ecosystem.\" 27 Oct 2026. https://astral.sh/blog/tar-codec (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://astral.sh/blog/tar-codec"},"body_markdown":"# tar-codec: a fast and secure tar parser for the Rust ecosystem\n\n*William Woodruff · Astral · October 27, 2026*\n\nStarting with 0.13[1](#user-content-fn-preview), [uv](https://github.com/astral-sh/uv) uses [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) to parse source distributions.\n\n[tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) is roughly twice as fast as our previous tar implementation ([astral-tokio-tar](https://crates.io/crates/astral-tokio-tar)) while\nalso reducing uv's exposure to parser differentials and other security risks.\n\nThis blog post is about [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) itself: what it is, and what compelled us to build and optimize\nit. Read on if you, too, would like to know too much about tar archives.\n\n![Performance of tar-codec relative to astral-tokio-tar, in uv's benchmarks](https://astral.sh/static/SVG/tar-codec-bench.svg)\n\n## Background: uv and tar\n\n*What follows is an extended digression on the tar format. If all you care about is `tar-codec`,\n[skip to it](#tar-codec)*.\n\nAs a Python packaging tool, uv is in the business of resolving and installing Python packages. [uv](https://github.com/astral-sh/uv)\nrequests *installable distributions* from various sources (principally package indices like [PyPI](https://pypi.org))\nand installs them on the user's behalf. This is how you go from `uv add urllib3` to something you\ncan actually import and use in your own Python codebase.\n\nInstallable distributions come in two primary forms:\n\n- \"Built\"[2](#user-content-fn-built) distributions, also known as *bdists* or (most commonly) *wheels*, are essentially\n ZIP archives containing structured metadata and a fixed layout. Installers handle these by (more\n or less) plopping their contents into the right destination directory.\n\n Many people think that *bdist* is short for \"binary distribution,\" since wheels often contain\n compiled native code. This one common way to use wheels, but wheels can also contain pure Python.\n They're just another (and frequently better) way to package and install a distribution.\n- \"Source\" distributions, also known as *sdists*, are essentially tar archives. Unlike built\n distributions, a source distribution is *essentially*[3](#user-content-fn-sdist) just a rollup of a source tree's\n state at release time.\n\n By definition, source distributions execute arbitrary code at installation time. This makes them\n very powerful, but brittle (missing build tools or dependencies) and also a security risk (an\n attacker who controls the sdist can run code without the user needing to `import` anything).\n\nWheels are generally produced by running a [build backend](https://peps.python.org/pep-0517/) on a source distribution. Examples of\nthat include uv's own [uv\\_build](https://docs.astral.sh/uv/concepts/build-backend/), [maturin](https://github.com/pyo3/maturin), and [flit](https://flit.pypa.io/en/stable/).\n\nConsequently, at a bare minimum, a Python package installer needs to understand at least two\ndifferent archive formats: ZIP for wheels, and tar for sdists.\n\nWe've already [talked a bit before](https://astral.sh/uv-security-advisory-cve-2025-54368) about ZIPs on the Astral blog, and we'll certainly be talking\nabout them again soon enough. But today we'll be focusing on tar.\n\n### Why a new tar parser?\n\nIt's fair to ask: why does [uv](https://github.com/astral-sh/uv) want a new tar parser?\n\nIn short: security and performance, but mostly security.\n\nUp until 0.13, uv used [astral-tokio-tar](https://crates.io/crates/astral-tokio-tar) for its tar operations, including both parsing and\ncreation (in `uv build`). [astral-tokio-tar](https://crates.io/crates/astral-tokio-tar) is itself a fork of [edera-dev/tokio-tar](https://github.com/edera-dev/tokio-tar), which is a\nfork of [vorot93/tokio-tar](https://github.com/vorot93/tokio-tar), which is a fork of [dignifiedquire/async-tar](https://github.com/dignifiedquire/async-tar), which is an async\n*adaptation* of [composefs/tar-rs](https://github.com/composefs/tar-rs).\n\nAround a year ago, we began to receive a [number of security reports](https://github.com/astral-sh/tokio-tar/security/advisories) in [astral-tokio-tar](https://crates.io/crates/astral-tokio-tar). This\nspurred us into looking more into our compression and archive handling stack, where we found\nadditional problems.\n\nMany of these boiled down to differentials that could cause a full parser\ndesynchronization[4](#user-content-fn-tarmageddon): given an identical tarball, [uv](https://github.com/astral-sh/uv), [pip](https://github.com/pypa/pip), and other package tooling\ncould be made to see different contents for the same package distribution. We found this\nparticularly concerning in the context of archives that would appear \"safe\" to an index (like PyPI)\nbut could extract dangerously when parsed by an installer like [uv](https://github.com/astral-sh/uv) or [pip](https://github.com/pypa/pip).\n\nThis could the take the form of an \"append\" differential (where the attacker can append new files or\nnew contents to existing files for a specific victim parser), or even a \"full\" differential (where\nthe attacker can fully separate the extraction contents for each parser):\n\nIn the context of [uv](https://github.com/astral-sh/uv), the attacker model for these kinds of vulnerabilities is somewhat subtle:\ntar is only used for source distributions which, per above and unlike wheels, are assumed to execute\narbitrary code at build time. Consequently, Python packaging *mostly* considers tar differentials to\nbe a minor concern, since any attacker who *would* use them has a much more direct and official\nroute towards writing arbitrary contents to disk (or executing arbitrary code).\n\nStill, these findings unnerved us, and we [embarked on a project](https://github.com/astral-sh/tokio-tar/pull/95) to eliminate every differential we\ncould find in [astral-tokio-tar](https://crates.io/crates/astral-tokio-tar).\n\n[tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) grew directly out of that effort: we ultimately came to the conclusion that the\ntokio-tar family was *too lenient by design*, and that retrofitting constraints to it would be more\ntroublesome and error-prone (including for existing non-Astral users of its APIs) than developing\nour own, ground-up implementation of tar. This would also give us the opportunity to systematically\nunderstand the tar format, and ensure that our assumptions around which aspects are (or aren't)\ndifferential risks hold true.\n\nSimilarly, we determined that our performance goals around tar parsing would be impossible to meet\nin the existing code base, at least not without refactors that would be tantamount to a rewrite.\n\n## (t)archaeology\n\nIt's important to understand that one does not simply build a tar parser from scratch. One must\nfirst understand *how* the tar format evolved into its current state[5](#user-content-fn-dorum).\n\ntar is an *interesting* format. Or more precisely, it's an interesting *family* of formats.\n\nTo summarize, there are three (and a half) variants that matter in practice:\n\n- There's UNIX V7 tar, which is the \"original\" tar. Very few tar implementations write this format\n by default anymore, but virtually all still read it. For example, Python's [`tarfile` module](https://docs.python.org/3/library/tarfile.html#supported-tar-formats) will\n happily parse V7 tar files.\n\n The UNIX V7 format is not widely used anymore because it comes with significant limitations on\n per-member file sizes (approximately 8 GiB) and path lengths (100 bytes).\n- Then there's the UStar/pax tar family[6](#user-content-fn-family). This family is widely supported by tar\n implementations for both reading and writing. As far as tar goes, UStar and pax are what passes\n for a \"gold standard\" in terms of clarity of specification and documentation.\n\n At this juncture, the main thing to know about the UStar/pax family is that it's derived from the\n UNIX V7 format, but with certain restrictions lifted:\n\n - UStar keeps the per-member size limit at 8 GiB, but raises the path length limit to 255 bytes\n (except for link names, which remain 100 bytes),\n - pax makes per-member sizes and path lengths unlimited, *including* link names.\n- Finally, there's GNU tar[7](#user-content-fn-oldgnu).\n\n GNU tar is documented in the [GNU tar manual](https://www.gnu.org/software/tar/manual/html_node/Standard.html), but is not standardized *per se*. The source code\n of GNU `tar` (the CLI) is the *de facto* standard for GNU tar.\n\n Like UStar/pax, GNU tar is derived from UNIX V7 tar. It's *also* derived from an [early draft of\n UStar](https://www.gnu.org/software/tar/manual/html_node/gnu.html).\n\n Also like pax, GNU tar makes file sizes and path/link lengths (effectively) unlimited. It does so\n in a completely different and much funnier way than pax does.\n\nThat gives us the following genealogy:\n\nIf you've ever written a parser for a binary format before, then you probably have a preconception\nfor what a binary format *should* look like. To first, second, and third approximations, binary\nformats tend to look *something* like this:\n\nWithin this, too, certain idioms appear: binary formats typically use some form of [TLV encoding](https://en.wikipedia.org/wiki/Type%E2%80%93length%E2%80%93value),\nuse different magics to signal different endiannesses, &c.\n\nThe tar format does not look like this, regardless of the variant. tar is fundamentally a *block\nformat*, owing to its origins as a tape archival format[8](#user-content-fn-tape).\n\nAs a block format, a tar archive is comprised of one or more fixed-length blocks. There is no\nfile-level header; instead, many of its blocks *are* headers, while other blocks contain actual\nmember[9](#user-content-fn-member) data:\n\nIn all flavors of tar, all blocks (including the header blocks) are always 512 bytes. Each header\nblock declares metadata for the *member* that follows it. This includes important things like the\nsize of the member and its filepath.\n\nMember contents don't always fit cleanly into multiples of 512 bytes, so a tar that wants to encode\na file member of `N` bytes will allocate `ceil(N / 512)` blocks and pad out the remainder of the\nlast block. This doesn't affect extraction, since the reader knows (via the header block) exactly\nhow many bytes out of the allocated space are actual file content.\n\nConventionally, the padding or \"holes\" in a tar archive are filled with NUL bytes. However, nothing\nstops a tar producer from filling them with uninitialized memory, or lines from the Bee Movie, or\nanything else. A recurring theme with tar (of all flavors) is that it has no opinions about what\ngoes in the holes of the format. This malleability makes it very prone to parser differentials,\nsince it gives the attacker ample space to place surprising data.\n\ntar's block design also makes it relatively space-inefficient[10](#user-content-fn-compression) by modern standards. For\nexample, a 513-byte file occupies an absolute minimum of 1,536 bytes when encoded as a tar member:\none 512-byte block for the header, and two blocks for the content (one of which has only 1 \"real\"\nbyte, followed by 511 bytes of padding). Similarly, a tar archive containing 1000 one-byte files\nrequires a minimum of 1025 KiB of encoding (1 MiB for the headers and contents, and then 1 KiB for\nthe trailing zero blocks).\n\n### tar problems\n\nSo far, this all sounds pretty sane. It's not a \"normal\" file format by modern standards, but\nthere's a simple state machine one could write to formalize a parse for the tar example above:\n\nUnfortunately, it's not that simple. The state machine above is accurate for a V7 tar, but not for\nanything else in the tar format family.\n\nTo understand why, we have to remember V7 tar's limitations. All of those limitations come down to\nthe C structure that represents a V7 member header, which we've annotated here:\n\n```\nstruct v7_header {\n // The member's path. Observe that this can be no more than 100 bytes;\n // NUL termination isn't required, since it's a fixed-size field.\n\tchar name[100];\n\n // The member's mode, as an octal string.\n // Note that only the first 4 octal digits (12 bits) have well-defined semantics in pax.\n\tchar mode[8];\n\n // The member's owner's UID and GID, as octal strings.\n\tchar uid[8];\n\tchar gid[8];\n\n // The size of the member, as an octal string.\n // V7 treats this as NUL-terminated, so the largest\n // interoperable value in it is `77777777777\\0`.\n\tchar size[12];\n\n // The member's mtime, as an octal string.\n\tchar mtime[12];\n\n // A checksum for the member header itself.\n\tchar chksum[8];\n\n // '\\0' for a regular file, '1' for a hard link,\n // and '2' for a symbolic link.\n\tchar linkflag;\n\n // When `linkflag` is '1' or '2', this is the link's destination.\n // Like `name` this is limited to 100 bytes.\n\tchar linkname[100];\n\n // 255 bytes of the header are unused.\n\tchar pad[255];\n};\n```\n\nThree big limitations stand out, and each member of the tar family tries to solve them in various\ndifferent ways:\n\n1. Member and link names cannot be longer than 100 bytes;\n2. Members cannot be larger than `0o77777777777` bytes, i.e. one byte short of 8 GiB;\n3. Only regular files and links are defined; there are no directory or other member types[11](#user-content-fn-dir).\n\nLet's see how each family works around these.\n\n### UStar and pax\n\nUStar's half-solution is to preserve the existing physical header block, while [impressing](https://en.wikipedia.org/wiki/Impressment) some of\nits fields and padding into new roles.\n\nThis is the UStar header:\n\n```\nstruct ustar_header {\n // Unchanged.\n\tchar name[100];\n\tchar mode[8];\n\tchar uid[8];\n\tchar gid[8];\n\tchar size[12];\n\tchar mtime[12];\n\tchar checksum[8];\n\n // `linkflag` is now `typeflag`, and encodes, inter alia,\n // '5' for a directory.\n\tchar typeflag[1];\n\n // Unchanged.\n\tchar linkname[100];\n\n // `\"ustar\\0\"`\n\tchar magic[6];\n\n // `\"00\"`\n\tchar version[2];\n\n // The user and group names for the member.\n\tchar uname[32];\n\tchar gname[32];\n\n // For chardev/blockdev members, the device's major and minor numbers.\n\tchar devmajor[8];\n\tchar devminor[8];\n\n // If nonempty, a prefix to prepend to `name` (joined by a `/`).\n\tchar prefix[155];\n\n // 12 remaining bytes of the header are unused.\n\tchar pad[12];\n};\n```\n\nThis partially lifts the member name restriction (by raising it to 255 bytes), but retains the file\nsize and linkname limitations.\n\npax fixes these remaining limitations by doing away with the single-block metadata design. Instead,\npax has *extensions* that rely on two new UStar typeflags:\n\n- Typeflag `'x'` indicates a *local* extended header block.\n- Typeflag `'g'` indicates a *global* extended header block.\n\nIn both cases, an extended header block indicates its member's \"size.\" But that size isn't for a\nfile: instead, it's for the extension data itself.\n\nSo, what's in the extension data? pax calls it an *extended header*, and it's an entirely different\nencoding format. An *extended header* contains one or more pax \"records,\" and each record is a\nstring that looks like this:\n\n```\n =\\n\n```\n\nwhere:\n\n- `` is the byte-length of the record itself, *including* `` itself[12](#user-content-fn-selfdesc);\n- `` is a keyword that sets or overrides some pax parsing state;\n- `` is the value to apply to ``.\n\nThis can be hard to visualize, so here's a concrete example:\n\n```\n13 size=8000↵\n```\n\n\"↵\" above indicates the linefeed.\n\nDespite being a new encoding, the way that pax records work is *relatively* simple: if there is an\n*effective* pax record that corresponds to a tar member header's state, the pax record's value takes\nprecedence. In the example above, this means that the subsequent member's size is `8000` bytes, even\nif the member's own header records a different size.\n\nThis is where the difference between global and local extensions comes in as well: global extended\nheaders take precedence over *all* subsequent members in the archive, while local extended headers\nonly affect the *next* file in the archive. Local extended headers also take precedence over any\nglobal headers, so we have the following precedence ordering:\n\n(This, too, is simplified. More on that later.)\n\nTo visualize all of that:\n\nFor this example, a pax-conforming tar implementation should extract two members:\n\n- `hasdrubal.txt`, with a payload of `10737418240` bytes and an `mtime` of `100`.\n\n Observe that this is both greater than the UStar size limit of 8 GiB and does *not* use the UStar\n header's own `name` (of `hannibal.txt`) or `size` (of `0`), since the member's header is preceded\n by an `x` extended header that overrides those values[13](#user-content-fn-own).\n\n Moreover, the archive *also* contains a `g` extended header that overrides `mtime` for all\n subsequent members and `x` does not contain its own `mtime`, so the global extended header's\n `mtime=100` also takes effect.\n- `hamilcar.txt`, with a payload of `1234` bytes and an `mtime` of `100`.\n\n Observe that, unlike `hasdrubal.txt`, there is no `x` extended header for this member. As a\n result, the UStar member states are used, with the exception of those overridden by the effective\n `g` extended header. Thus `hamilcar.txt` ends up with `mtime=100`, even if its UStar header sets a\n different value.\n\n### GNU tar\n\nGNU tar takes a very different approach. First, here's the GNU tar header:\n\n```\nstruct gnu_header {\n // Unchanged from UStar.\n\tchar name[100];\n\tchar mode[8];\n\n // Same layout as UStar, but potentially base-256.\n\tchar uid[8];\n\tchar gid[8];\n\tchar size[12];\n\tchar mtime[12];\n\tchar checksum[8];\n\n // Same layout as UStar, but additional typeflags.\n\tchar typeflag[1];\n\n // Unchanged.\n\tchar linkname[100];\n\n // `\"ustar \"`\n\tchar magic[6];\n\n // `\" \\0\"`\n\tchar version[2];\n\n // Unchanged from UStar.\n\tchar uname[32];\n\tchar gname[32];\n\tchar devmajor[8];\n\tchar devminor[8];\n\n // GNU extensions.\n\tchar atime[12];\n\tchar ctime[12];\n\tchar offset[12];\n\tchar longnames[4];\n\tchar unused[1];\n\tstruct {\n\t\tchar offset[12];\n\t\tchar numbytes[12];\n\t} sparse[4];\n\tchar isextended[1];\n\tchar realsize[12];\n\n // 17 remaining bytes of the header are unused.\n\tchar pad[17];\n};\n```\n\nObserve that, unlike the UStar format, there is no `char prefix[155]` field. Furthermore, instead of\npax extended headers with a fully general keyword/value format, GNU tar does two things:\n\n- It goes *way* harder than UStar on trying to re-use the existing 512-byte V7 member header. In\n particular, GNU tar adds its own base-256 encoding: if the first byte in a numeric field has its\n high bit set, then the remaining octets in that field are binary, rather than tar's normal octal\n form.\n\n This effectively expands the per-byte entropy in numeric fields from 3 bits to 8 bits. For a\n `char size[12]`, that makes the effective size limit `2**80`[14](#user-content-fn-big) or one yobibyte.\n- Instead of general `x` and `g` extended headers, GNU tar introduces a bunch of *specialized*\n typeflags for its own extensions.\n\n [Many of these](https://www.gnu.org/software/tar/manual/html_node/Standard.html) are not relevant to the post, or even to modern tar archives at all. But two are:\n\n - The `L` typeflag identifies the next sequence of content blocks as containing a filename.\n - The `K` typeflag identifies the next sequence of content blocks as containing a link name.\n\n These two typeflags effectively fill the role of the pax `path` and `linkpath` records,\n respectively.\n\nTo visualize that:\n\nFor this example, a GNU tar implementation should extract two members:\n\n- `hasdrubal.txt` with a payload of `10737418240` bytes.\n\n Observe that this size, which exceeds 8 GiB, was encoded directly in the header. However, the\n effective filename of `hasdrubal.txt` comes from overriding the header's original filename of\n `hannibal.txt` via the preceding GNU `L` extension.\n- `publius.txt`, which links to another file named `africanus.txt`.\n\n Observe that `publius.txt` originally declared `publius.txt -> scipio.txt` as its link, but this\n gets overridden to `africanus.txt` by the preceding GNU `K` extension.\n\nFinally, while GNU tar does not have pax's notion of global vs. local precedence, GNU tar extensions\ncan *stack*. For example, to override a member's `name` and `linkname` at the same time, you'd have\na tar header sequence like this[15](#user-content-fn-order):\n\n## tar-codec\n\n### State machine [turducken](https://en.wikipedia.org/wiki/Turducken)\n\nNow that we understand *very roughly* how the various tar family formats look and work, let's see\nwhat [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) has to do to parse them safely and efficiently.\n\nFirst we'll revisit the state machine. Our nice and simple V7 state machine now understands the\ndifferent pax and GNU states, and has become this monstrosity:\n\n(Note that the error transitions in this state machine are dramatically simplified: in practice, any\nI/O or validation failure at any point also results in a `Failed` state.)\n\nWhat in the world happened? A few different things:\n\n- Our state machine begins with a \"classification\" stage, which effectively locks the remainder of\n the parse to a single family (either UStar/pax or GNU[16](#user-content-fn-nov7)) based on the identity (magic and\n version) of the first block.\n\n This is done for differential reduction reasons: while researching [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec)'s design, we\n realized that many other widely used tar parsers attempt to merge states across various tar\n families, with inconsistent results.\n\n For example, with some tar parsers, it's possible to stack headers like so:\n\n This has unclear semantics: GNU tar expects an `L` or `K` to apply to the next \"file\" member, but\n the next physical member in the stream is a pax local extension that *also* wants to modify the\n next \"file\" member. In practice we've seen other parsers do all kinds of things in this state,\n including attempting to merge the extensions, honoring one over the other(s), attempting to apply\n the `L` state to `x` instead of the file member, and so forth. Each of these is a potentially\n exploitable differential!\n\n Instead, [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) will reject a tar archive if its individual headers don't agree about what\n family of tar it is. Real-world tar encoders don't mix tar families (there's no good reason to),\n so this doesn't pose a significant breakage risk.\n- The GNU part of the state machine is slightly more strict than the GNU documentation implies: the\n GNU documentation allows stacking of `L`/`K` extensions, including *multiple of the same kind*, in\n any order.\n\n [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) tightens that to allowing `L` and `K` in any order, but only one of each per\n underlying member.\n- The UStar/pax part of the state machine models local and global extended headers rigidly.\n\n There's a significant amount of complexity beneath this: pax allows `x` and `g` *anywhere* in the\n tar stream. For example, this is a perfectly valid extension sequence in pax:\n\n This brings us to one of the most painful parts of pax that we glossed over before: extended\n headers don't *shadow* each other fully, they *compose* depending on what state the \"stack\" of\n effective pax records is in. The state of any given pax record can be:\n\n - *Undeclared*: no effective pax record exists anywhere in the current stack of local and global\n extensions, which means that the member's own UStar header's state takes precedence.\n - *Active*: there's an effective pax record in the stack, *and* it has a value.\n - *Tombstoned*: there's an effective pax record in the stack, but it has no value (for example,\n `8 path=\\n` instead of `12 path=abc\\n`).\n\n You would think that this would be the same as the *Undeclared* state, but it's not! pax\n considers this record **explicitly** deleted, and [prescribes](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pax.html#:~:text=If%20the%20%3Cvalue%3E%20field%20is%20zero%20length%2C%20it%20shall%20delete%20any%20header%20block%20field%2C%20previously%20entered%20extended%20header%20value%2C%20or%20global%20extended%20header%20value%20of%20the%20same%20name.):\n\n > If the `` field is zero length, it shall delete any header block field, previously\n > entered extended header value, or global extended header value of the same name.\n\n This has fascinating consequences for the state machine, because it means that an\n effective-but-deleted `path` record means...a file with no path? How are you supposed to extract\n that[17](#user-content-fn-random)?\n\n Needless to say, [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) tries to follow this faithfully. In particular, it attempts to be\n diligent about rejecting logically-consistent-but-nonsense states that fall out of pax's strange\n record state machine, including empty paths and global extended headers that try to declare `size`\n and other key states for all members.\n\n### From parsing to extraction\n\nOne of the early design choices we made with [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) was to isolate the implementation into a\nfew layers. From lowest to highest:\n\n- The [tar-framing](https://crates.io/crates/tar-framing) crate is responsible for taking a stream of blocks and turning it into a stream\n of frames. This happens in two refinement stages:\n\n - The stream or \"physical\" API honors the big state machine above and yields each block in a given\n state, one block at a time.\n - The \"logical\" API consumes the blocks produced by the \"physical\" API, and yields them as a\n stream of assembled members. Each assembled member is essentially its \"header\" (which is the\n physical header, merged with any effective extension state that affected the physical framing),\n a separate extension state for non-framing extensions (such as GNU `L` and `K`), and a readable\n payload for the member's actual contents.\n- The [tar-codec](https://crates.io/crates/tar-codec) crate takes the \"logical\" API from [tar-framing](https://crates.io/crates/tar-framing) and turns it\n into something that can be extracted in memory or onto disk.\n\nExtracting an archive (of any format) onto disk is its own can of worms, with its own differential\nrisks[18](#user-content-fn-extdiff). Another one of our early observations is that every archive implementation tends to\nhave its own extract-to-disk implementation, with its own edge cases.\n\nWe built [archive-trait](https://crates.io/crates/archive-trait) to address that. [archive-trait](https://crates.io/crates/archive-trait) exposes `Archive` and `ArchiveBuilder`\ntraits (along with associated types and machinery) that allow any format to expose uniform APIs for\nreading and creating archives. We then built [archive-trait](https://crates.io/crates/archive-trait)'s own filesystem extraction on top of\n[cap-std](https://crates.io/crates/cap-std), giving us the ability to \"anchor\" extraction into a single directory and avoid a large\nclass of link and path traversal footguns.\n\n## Lessons learned\n\n- Writing a differential-resistant tar parser is *surprisingly* hard, given how ubiquitous tar files\n of all flavors are, and how simple the original (V7) format is.\n\n This post has only scratched the surface in terms of complexity. To recap:\n\n - Each tar family has its own state machine, which is not always clearly laid out in that family's documentation\n or reference implementation. It is *imperative* that these state machines do not mix, but real-world tar\n parsers regularly allow them to.\n - Each tar family has its own, non-trivial rules for determining the precedence of state (pathname, linkname, and size)\n attached to archive members.\n\n Some of this state affects the framing of the parser itself, meaning that faithfully\n handling e.g. pax archives means handling `size` and similar records up-front, rather than trying to abstract the\n parse across individual 512-byte header blocks.\n\n Separately, there were numerous places where we had to compromise on strict pax conformance\n because of observed behavior from other implementations:\n\n - The pax spec is explicit that pax record values must be UTF-8. However, both GNU tar and `star` historically emitted\n some vendor-specific records[19](#user-content-fn-paxvendor) as raw binary. See [this bug-tar thread for more context](https://lists.gnu.org/r/bug-tar/2013-12/msg00012.html).\n - UStar and pax are explicit that `uname` and `gname` should be NUL-terminated in the UStar header, meaning that each can\n be at most 31 bytes long. However, CPython's `tarfile` (and probably others) will happily fill either to 32 bytes, only\n overflowing to the pax record at 33 bytes or more. See [CPython#159322](https://github.com/python/cpython/issues/159322) for more context.\n - More generally, tar implementations will frequently leave `mode`, `uid`, `gid`, and `mtime` as all NUL, rather than\n zeroed or similar. This doesn't pose a framing/parser synchronization risk, so [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) tracks them as \"missing\"\n rather than assigning an arbitrary default value.\n\n Finally, in the process of designing and implementing tar-codec, we found *many* parser\n differential primitives that cause various tar implementations to behave *very* differently on\n identical inputs. Some of these are currently being disclosed as security vulnerabilities.\n However, some of the general classes we saw:\n\n - pax/GNU state machine confusion, as mentioned above. In particular, some tar implementations can be fooled\n into applying a pax extension's states to a GNU extension (or vice versa) instead of to the subsequent\n member, often resulting in full desynchronization and different extraction results.\n - Mis-handling of pax extension precedence, in particular both global (`g`) extended headers and\n record \"tombstoning.\" For example, some implementations will accept a `g` header with `size=1234`\n but ignore it, despite pax saying clearly that such a `size` record overrides any UStar `size` fields.\n This also typically results in a full desynchronization primitive.\n - Mis-handling of pax record parsing. For example, some implementations will ignore the ``\n prefix and instead assume that pax records can be safely split on `\\n`. This is not true, since\n the `` of a pax record can contain a verbatim `\\n`. As you may have guessed, this also\n typically results in a full desynchronization primitive.\n - More generally, we found a great deal of variety in how lax various tar implementations are when it comes to parsing\n integer fields: many use their language's native \"integer from string\" API, which can [behave surprisingly](https://yossarian.net/til/post/some-surprising-numeric-conversions-in-python/)\n in the presence of different (non-Western Arabic) numeral forms, leading and trailing whitespace,\n and so forth. These small variations can percolate into two different implementations parsing\n the same `size` field as different integers, once again resulting in desynchronization.\n- Decode and encode performance was *not* the hard part: we found a lot of low-hanging fruit while\n optimizing both, including optimizing tar's checksum generation and calculating pax record length\n prefixes.\n\n We also picked up significant performance gains by ensuring that tar-codec reuses allocations and\n batches/buffers its work wherever possible (and where doing so does not weaken the API's internal\n consistency or cancellation correctness in an `async` setting).\n- Conversely, *performant* secure extraction *is* somewhat hard: directory capability tracking (of\n the kind that [cap-std](https://crates.io/crates/cap-std) does) adds measurable overhead to many of our filesystem operations,\n particularly when extracting on hosts that lack [`openat2(2)`](https://man7.org/linux/man-pages/man2/openat2.2.html) or an equivalent system\n call[20](#user-content-fn-openat). This negatively impacts extraction on macOS in particular.\n\nAll in all, [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec)'s design rests on well-worn principles for secure parser design: we are\n[conservative rather than liberal](https://alexgaynor.net/2025/mar/25/postels-law-and-the-three-ring-circus/) in what we accept, and we enforce correctness by construction\nthrough increasingly constrictive layers (from member framing to streaming to extraction). These\ndesign choices have made us confident that [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) is more resistant to differentials (by\ncorrectly rejecting them) than other tar implementations.\n\nThe fractured and under-specified nature of tar makes it impossible for us to say that we are\n*entirely* devoid of differentials, or that new differentials won't be discovered in the future.\nHowever, we believe that [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec)'s primitives are the correct ones if and when those\ncontingencies occur.\n\n## Looking forwards\n\nThis is not the end of our work, on several levels:\n\n- While [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) is essentially feature-complete for our own purposes (within [uv](https://github.com/astral-sh/uv)), we're\n interested in making sure that it also solves problems that the larger Open Source community has\n around secure and performant tar parsing.\n\n To that end, we'll be looking at supporting other, more niche aspects of the various tar families\n (like sparse files, and more control over vendor-specific pax records) as the need arises.\n\n Separately, we'll be working with others in the community on improving the correctness of *other*\n tar implementations. In particular, we see Harmen Stoppels's nascent [tar-minefield](https://stoppels.ch/tar/) project as a\n promising way to test and re-align other implementations with stricter, more\n differential-resistant interpretations of the tar family formats.\n- Separately, as you may have guessed from our decision to build a generic [archive-trait](https://crates.io/crates/archive-trait) library:\n tar is not the only format we're thinking about here!\n\n We're also actively looking into replacing uv's ZIP parsing, for similar reasons. We hope to share\n our lessons from that work soon, along with material benefits to uv itself.\n\n## Footnotes\n\n1. [tar-codec](https://github.com/astral-sh/astral-codec/tree/main/crates/tar-codec) has been available since 0.12, but was disabled by default behind the\n `--preview-features tar-codec` flag. As of 0.13, it is the default and can be disabled by\n setting `UV_LEGACY_TAR_BACKEND=1`. [↩](#user-content-fnref-preview)\n2. The term \"built\" is the source of a great deal of confusion for newcomers when they first\n encounter Python packaging. In the context of Python packaging, \"built\" refers to the fact that\n the distribution is produced by a \"build backend\", i.e. contains structured metadata that a\n source distribution is not guaranteed to have.\n\n It does **not** imply that the wheel contains shared libraries or other native code, although it can.\n Pure Python packages are also regularly distributed as built distributions, because installing them is\n faster and more reliable. [↩](#user-content-fnref-built)\n3. Historically, a source distribution was *literally* just a source archive, with the convention\n of having a `pyproject.toml` or `setup.py` at the archive's root. More recent Python packaging\n standards state that source distributions must contain a `PKG-INFO` file containing standard\n metadata. See [PEP 643](https://peps.python.org/pep-0643/) for more on that.\n\n As a result, many source distributions contain no metadata other than the build system itself, which must be\n executed to access the distribution's metadata. [↩](#user-content-fnref-sdist)\n4. We recommend Edera's \"[TARmageddon](https://edera.dev/stories/tarmageddon)\" blog post for a detailed treatment of one such\n differential. [↩](#user-content-fnref-tarmageddon)\n5. [Martin Dørum](https://mort.coffee) has a [great blog post](https://mort.coffee/home/tar/) on the original tar format and how it evolved over time\n into the constellation of variants that currently exist. We highly recommend reading his post. [↩](#user-content-fnref-dorum)\n6. Whether or not UStar and pax should be clumped together in the same family is doubtless\n extremely controversial to the dozen-or-so other people on Earth who have thought about the tar\n format this much. I consider them to be in the same family, since pax is *effectively* a strict\n superset of UStar and comes from the same standards body (POSIX) that produced UStar. [↩](#user-content-fnref-family)\n7. There are actually two different GNU tar formats: there's `oldgnu` i.e. \"old GNU\" and `gnu` i.e.\n \"new GNU\". The difference between these two is minor (relative to everything else) and GNU tar\n itself emits \"new GNU\" as of tar 1.12, which was released in 1997. So that's enough about \"old\n GNU.\" [↩](#user-content-fnref-oldgnu)\n8. \"tar\" of course being short for \"Tape ARchive.\" Magnetic tapes (like all mass storage) store\n their contents in sequences of fixed-sized blocks, and tar seals 1979's glorious 512-byte blocks\n in amber.\n\n tar is also far from the only block-style file format. [cpio](https://en.wikipedia.org/wiki/Cpio) is another well-known one. [↩](#user-content-fnref-tape)\n9. Archive members are things like files and directories, but *also* things like sockets and device\n files. I'm not really clear on what it means to archive a socket, but both UStar/pax and GNU tar\n will happily let you express that you've done so! [↩](#user-content-fnref-member)\n10. Before compression, of course. tar knows nothing about compression. It's just the format that\n encodes one or more files into a single file. [↩](#user-content-fnref-compression)\n11. V7 tar handles directories in two ways: it creates directories that are *implied* to exist (for\n example, a member whose name is `foo/makeme/bar` will create `foo` and `foo/makeme` if needed),\n and it will explicitly create a directory if a regular file member ends with a slash, e.g.\n `foo/makeme/`. [↩](#user-content-fnref-dir)\n12. In other words, it's a self-describing length. This is somewhat unusual when compared to a\n \"normal\" TLV/LTV encoding, where the length component is typically either a fixed length or\n something with only limited length variation, like [LEB128](https://en.wikipedia.org/wiki/LEB128). [↩](#user-content-fnref-selfdesc)\n13. You might be thinking: why bother putting states in the member's UStar header at all, if they're\n just going to be overridden by the pax extended header? Why not just fill those parts of the\n block with NULs?\n\n The answer has to do with backwards compatibility: both GNU and pax do a ~~disgusting hack~~\n interesting trick where the extension/extended header's *own* name field is also a valid\n path, conventionally a specially formatted one like `%d/PaxHeaders.%p/%f`, where `%d` is\n the dirname of the pax `path` record, `%p` is the current PID, and `%f` is the basename\n of the same `path` record.\n\n The theory behind this is (probably) that a subsequent tool can repair the extraction,\n by looking at the pax headers which have been extracted as a regular file instead.\n In practice I suspect this doesn't work well in the greater-than-8 GiB case, which\n is one of the cases where it matters the most. [↩](#user-content-fnref-own)\n14. This is `2**80` instead of `2**96` because we lose `size[0]` to the base-256 sign bit and\n `size[11]` to the NUL byte. Implementations that don't require a NUL terminator for fields like\n `size` could in principle allow up to `2**88` instead. [↩](#user-content-fnref-big)\n15. The order of GNU extensions before the member they affect does not appear to matter in any\n implementation. In other words, you can have `L -> K -> 1` or `K -> L -> 1`. [↩](#user-content-fnref-order)\n16. That's correct: we don't support parsing V7-style tar files. An early review revealed that\n they're nonexistent in the Python packaging ecosystem, probably because Python's own `tarfile`\n never supported creating them. The gap between V7 and UStar is small, however, so adding it\n would not be difficult *if* needed. [↩](#user-content-fnref-nov7)\n17. Many tar implementations do random things here, like trying to extract to an empty path (which\n of course fails), or picking *some* previous filename or extended header state. [↩](#user-content-fnref-random)\n18. Extraction differentials themselves are out of this post's scope, but to summarize: extraction\n depends heavily on the target filesystem and its state, including any pre-existing symbolic and\n hard links.\n\n Nuances abound, including how to handle files that differ in casing inside the archive but not on the filesystem itself\n (macOS and Windows), how to handle symlinking on systems where symbolic links are privileged (Windows), and how to handle \"funky but valid\" filenames\n (non-UTF-8, UTF-8 with NULs, control characters, &c.). [↩](#user-content-fnref-extdiff)\n19. A pax \"vendor record\" is a normal pax record, but with a name like `VENDOR.keyword` instead of\n `keyword`. pax itself does not specify their semantics and various implementations do wildly\n different things with them, so tar-codec rejects them by default (with an option to skip/ignore\n them instead). `SCHILY.xattr.*` is one common example of a vendor record namespace. GNU also has\n their own vendor namespace, e.g. `GNU.sparse.*` for sparse members. [↩](#user-content-fnref-paxvendor)\n20. Without `openat2(2)` or a similar source of `RESOLVE_BENEATH` semantics, `cap-std` needs to open\n each directory component in a path, which means that its system calls scale linearly with the\n depth of the paths it needs to track. This is on top of the system call overhead for the\n extraction I/O itself. [↩](#user-content-fnref-openat)\n\n---\n\n*Originally published at [Astral](https://astral.sh/blog/tar-codec).*\n","body_html":"

tar-codec: a fast and secure tar parser for the Rust ecosystem

\n

William Woodruff · Astral · October 27, 2026

\n

Starting with 0.131, uv uses tar-codec to parse source distributions.

\n

tar-codec is roughly twice as fast as our previous tar implementation (astral-tokio-tar) while\nalso reducing uv's exposure to parser differentials and other security risks.

\n

This blog post is about tar-codec itself: what it is, and what compelled us to build and optimize\nit. Read on if you, too, would like to know too much about tar archives.

\n
\"Performance
\n

Background: uv and tar

\n

*What follows is an extended digression on the tar format. If all you care about is tar-codec,\nskip to it*.

\n

As a Python packaging tool, uv is in the business of resolving and installing Python packages. uv\nrequests installable distributions from various sources (principally package indices like PyPI)\nand installs them on the user's behalf. This is how you go from uv add urllib3 to something you\ncan actually import and use in your own Python codebase.

\n

Installable distributions come in two primary forms:

\n

  • "Built"2 distributions, also known as bdists or (most commonly) wheels, are essentially

    ZIP archives containing structured metadata and a fixed layout. Installers handle these by (more\nor less) plopping their contents into the right destination directory.

    \n

    Many people think that bdist is short for "binary distribution," since wheels often contain\ncompiled native code. This one common way to use wheels, but wheels can also contain pure Python.\nThey're just another (and frequently better) way to package and install a distribution.

  • "Source" distributions, also known as sdists, are essentially tar archives. Unlike built

    distributions, a source distribution is essentially3 just a rollup of a source tree's\nstate at release time.

    \n

    By definition, source distributions execute arbitrary code at installation time. This makes them\nvery powerful, but brittle (missing build tools or dependencies) and also a security risk (an\nattacker who controls the sdist can run code without the user needing to import anything).

\n

Wheels are generally produced by running a build backend on a source distribution. Examples of\nthat include uv's own uv_build, maturin, and flit.

\n

Consequently, at a bare minimum, a Python package installer needs to understand at least two\ndifferent archive formats: ZIP for wheels, and tar for sdists.

\n

We've already talked a bit before about ZIPs on the Astral blog, and we'll certainly be talking\nabout them again soon enough. But today we'll be focusing on tar.

\n

Why a new tar parser?

\n

It's fair to ask: why does uv want a new tar parser?

\n

In short: security and performance, but mostly security.

\n

Up until 0.13, uv used astral-tokio-tar for its tar operations, including both parsing and\ncreation (in uv build). astral-tokio-tar is itself a fork of edera-dev/tokio-tar, which is a\nfork of vorot93/tokio-tar, which is a fork of dignifiedquire/async-tar, which is an async\nadaptation of composefs/tar-rs.

\n

Around a year ago, we began to receive a number of security reports in astral-tokio-tar. This\nspurred us into looking more into our compression and archive handling stack, where we found\nadditional problems.

\n

Many of these boiled down to differentials that could cause a full parser\ndesynchronization4: given an identical tarball, uv, pip, and other package tooling\ncould be made to see different contents for the same package distribution. We found this\nparticularly concerning in the context of archives that would appear "safe" to an index (like PyPI)\nbut could extract dangerously when parsed by an installer like uv or pip.

\n

This could the take the form of an "append" differential (where the attacker can append new files or\nnew contents to existing files for a specific victim parser), or even a "full" differential (where\nthe attacker can fully separate the extraction contents for each parser):

\n

In the context of uv, the attacker model for these kinds of vulnerabilities is somewhat subtle:\ntar is only used for source distributions which, per above and unlike wheels, are assumed to execute\narbitrary code at build time. Consequently, Python packaging mostly considers tar differentials to\nbe a minor concern, since any attacker who would use them has a much more direct and official\nroute towards writing arbitrary contents to disk (or executing arbitrary code).

\n

Still, these findings unnerved us, and we embarked on a project to eliminate every differential we\ncould find in astral-tokio-tar.

\n

tar-codec grew directly out of that effort: we ultimately came to the conclusion that the\ntokio-tar family was too lenient by design, and that retrofitting constraints to it would be more\ntroublesome and error-prone (including for existing non-Astral users of its APIs) than developing\nour own, ground-up implementation of tar. This would also give us the opportunity to systematically\nunderstand the tar format, and ensure that our assumptions around which aspects are (or aren't)\ndifferential risks hold true.

\n

Similarly, we determined that our performance goals around tar parsing would be impossible to meet\nin the existing code base, at least not without refactors that would be tantamount to a rewrite.

\n

(t)archaeology

\n

It's important to understand that one does not simply build a tar parser from scratch. One must\nfirst understand how the tar format evolved into its current state5.

\n

tar is an interesting format. Or more precisely, it's an interesting family of formats.

\n

To summarize, there are three (and a half) variants that matter in practice:

\n

  • There's UNIX V7 tar, which is the "original" tar. Very few tar implementations write this format

    by default anymore, but virtually all still read it. For example, Python's tarfile module will\nhappily parse V7 tar files.

    \n

    The UNIX V7 format is not widely used anymore because it comes with significant limitations on\nper-member file sizes (approximately 8 GiB) and path lengths (100 bytes).

  • Then there's the UStar/pax tar family6. This family is widely supported by tar

    implementations for both reading and writing. As far as tar goes, UStar and pax are what passes\nfor a "gold standard" in terms of clarity of specification and documentation.

    \n

    At this juncture, the main thing to know about the UStar/pax family is that it's derived from the\nUNIX V7 format, but with certain restrictions lifted:

    \n

    • UStar keeps the per-member size limit at 8 GiB, but raises the path length limit to 255 bytes

      (except for link names, which remain 100 bytes),

    • pax makes per-member sizes and path lengths unlimited, including link names.

  • Finally, there's GNU tar7.

    GNU tar is documented in the GNU tar manual, but is not standardized per se. The source code\nof GNU tar (the CLI) is the de facto standard for GNU tar.

    \n

    Like UStar/pax, GNU tar is derived from UNIX V7 tar. It's also derived from an early draft of\nUStar.

    \n

    Also like pax, GNU tar makes file sizes and path/link lengths (effectively) unlimited. It does so\nin a completely different and much funnier way than pax does.

\n

That gives us the following genealogy:

\n

If you've ever written a parser for a binary format before, then you probably have a preconception\nfor what a binary format should look like. To first, second, and third approximations, binary\nformats tend to look something like this:

\n

Within this, too, certain idioms appear: binary formats typically use some form of TLV encoding,\nuse different magics to signal different endiannesses, &c.

\n

The tar format does not look like this, regardless of the variant. tar is fundamentally a *block\nformat*, owing to its origins as a tape archival format8.

\n

As a block format, a tar archive is comprised of one or more fixed-length blocks. There is no\nfile-level header; instead, many of its blocks are headers, while other blocks contain actual\nmember9 data:

\n

In all flavors of tar, all blocks (including the header blocks) are always 512 bytes. Each header\nblock declares metadata for the member that follows it. This includes important things like the\nsize of the member and its filepath.

\n

Member contents don't always fit cleanly into multiples of 512 bytes, so a tar that wants to encode\na file member of N bytes will allocate ceil(N / 512) blocks and pad out the remainder of the\nlast block. This doesn't affect extraction, since the reader knows (via the header block) exactly\nhow many bytes out of the allocated space are actual file content.

\n

Conventionally, the padding or "holes" in a tar archive are filled with NUL bytes. However, nothing\nstops a tar producer from filling them with uninitialized memory, or lines from the Bee Movie, or\nanything else. A recurring theme with tar (of all flavors) is that it has no opinions about what\ngoes in the holes of the format. This malleability makes it very prone to parser differentials,\nsince it gives the attacker ample space to place surprising data.

\n

tar's block design also makes it relatively space-inefficient10 by modern standards. For\nexample, a 513-byte file occupies an absolute minimum of 1,536 bytes when encoded as a tar member:\none 512-byte block for the header, and two blocks for the content (one of which has only 1 "real"\nbyte, followed by 511 bytes of padding). Similarly, a tar archive containing 1000 one-byte files\nrequires a minimum of 1025 KiB of encoding (1 MiB for the headers and contents, and then 1 KiB for\nthe trailing zero blocks).

\n

tar problems

\n

So far, this all sounds pretty sane. It's not a "normal" file format by modern standards, but\nthere's a simple state machine one could write to formalize a parse for the tar example above:

\n

Unfortunately, it's not that simple. The state machine above is accurate for a V7 tar, but not for\nanything else in the tar format family.

\n

To understand why, we have to remember V7 tar's limitations. All of those limitations come down to\nthe C structure that represents a V7 member header, which we've annotated here:

\n
struct v7_header {\n    // The member's path. Observe that this can be no more than 100 bytes;\n    // NUL termination isn't required, since it's a fixed-size field.\n    char name[100];\n\n    // The member's mode, as an octal string.\n    // Note that only the first 4 octal digits (12 bits) have well-defined semantics in pax.\n    char mode[8];\n\n    // The member's owner's UID and GID, as octal strings.\n    char uid[8];\n    char gid[8];\n\n    // The size of the member, as an octal string.\n    // V7 treats this as NUL-terminated, so the largest\n    // interoperable value in it is `77777777777\\0`.\n    char size[12];\n\n    // The member's mtime, as an octal string.\n    char mtime[12];\n\n    // A checksum for the member header itself.\n    char chksum[8];\n\n    // '\\0' for a regular file, '1' for a hard link,\n    // and '2' for a symbolic link.\n    char linkflag;\n\n    // When `linkflag` is '1' or '2', this is the link's destination.\n    // Like `name` this is limited to 100 bytes.\n    char linkname[100];\n\n    // 255 bytes of the header are unused.\n    char pad[255];\n};
\n

Three big limitations stand out, and each member of the tar family tries to solve them in various\ndifferent ways:

\n
  1. Member and link names cannot be longer than 100 bytes;
  2. Members cannot be larger than 0o77777777777 bytes, i.e. one byte short of 8 GiB;
  3. Only regular files and links are defined; there are no directory or other member types11.
\n

Let's see how each family works around these.

\n

UStar and pax

\n

UStar's half-solution is to preserve the existing physical header block, while impressing some of\nits fields and padding into new roles.

\n

This is the UStar header:

\n
struct ustar_header {\n    // Unchanged.\n    char name[100];\n    char mode[8];\n    char uid[8];\n    char gid[8];\n    char size[12];\n    char mtime[12];\n    char checksum[8];\n\n    // `linkflag` is now `typeflag`, and encodes, inter alia,\n    // '5' for a directory.\n    char typeflag[1];\n\n    // Unchanged.\n    char linkname[100];\n\n    // `"ustar\\0"`\n    char magic[6];\n\n    // `"00"`\n    char version[2];\n\n    // The user and group names for the member.\n    char uname[32];\n    char gname[32];\n\n    // For chardev/blockdev members, the device's major and minor numbers.\n    char devmajor[8];\n    char devminor[8];\n\n    // If nonempty, a prefix to prepend to `name` (joined by a `/`).\n    char prefix[155];\n\n    // 12 remaining bytes of the header are unused.\n    char pad[12];\n};
\n

This partially lifts the member name restriction (by raising it to 255 bytes), but retains the file\nsize and linkname limitations.

\n

pax fixes these remaining limitations by doing away with the single-block metadata design. Instead,\npax has extensions that rely on two new UStar typeflags:

\n
  • Typeflag 'x' indicates a local extended header block.
  • Typeflag 'g' indicates a global extended header block.
\n

In both cases, an extended header block indicates its member's "size." But that size isn't for a\nfile: instead, it's for the extension data itself.

\n

So, what's in the extension data? pax calls it an extended header, and it's an entirely different\nencoding format. An extended header contains one or more pax "records," and each record is a\nstring that looks like this:

\n
<length> <keyword>=<value>\\n
\n

where:

\n
  • <length> is the byte-length of the record itself, including <length> itself12;
  • <keyword> is a keyword that sets or overrides some pax parsing state;
  • <value> is the value to apply to <keyword>.
\n

This can be hard to visualize, so here's a concrete example:

\n
13 size=8000↵
\n

"↵" above indicates the linefeed.

\n

Despite being a new encoding, the way that pax records work is relatively simple: if there is an\neffective pax record that corresponds to a tar member header's state, the pax record's value takes\nprecedence. In the example above, this means that the subsequent member's size is 8000 bytes, even\nif the member's own header records a different size.

\n

This is where the difference between global and local extensions comes in as well: global extended\nheaders take precedence over all subsequent members in the archive, while local extended headers\nonly affect the next file in the archive. Local extended headers also take precedence over any\nglobal headers, so we have the following precedence ordering:

\n

(This, too, is simplified. More on that later.)

\n

To visualize all of that:

\n

For this example, a pax-conforming tar implementation should extract two members:

\n

  • hasdrubal.txt, with a payload of 10737418240 bytes and an mtime of 100.

    Observe that this is both greater than the UStar size limit of 8 GiB and does not use the UStar\nheader's own name (of hannibal.txt) or size (of 0), since the member's header is preceded\nby an x extended header that overrides those values13.

    \n

    Moreover, the archive also contains a g extended header that overrides mtime for all\nsubsequent members and x does not contain its own mtime, so the global extended header's\nmtime=100 also takes effect.

  • hamilcar.txt, with a payload of 1234 bytes and an mtime of 100.

    Observe that, unlike hasdrubal.txt, there is no x extended header for this member. As a\nresult, the UStar member states are used, with the exception of those overridden by the effective\ng extended header. Thus hamilcar.txt ends up with mtime=100, even if its UStar header sets a\ndifferent value.

\n

GNU tar

\n

GNU tar takes a very different approach. First, here's the GNU tar header:

\n
struct gnu_header {\n    // Unchanged from UStar.\n    char name[100];\n    char mode[8];\n\n    // Same layout as UStar, but potentially base-256.\n    char uid[8];\n    char gid[8];\n    char size[12];\n    char mtime[12];\n    char checksum[8];\n\n    // Same layout as UStar, but additional typeflags.\n    char typeflag[1];\n\n    // Unchanged.\n    char linkname[100];\n\n    // `"ustar "`\n    char magic[6];\n\n    // `" \\0"`\n    char version[2];\n\n    // Unchanged from UStar.\n    char uname[32];\n    char gname[32];\n    char devmajor[8];\n    char devminor[8];\n\n    // GNU extensions.\n    char atime[12];\n    char ctime[12];\n    char offset[12];\n    char longnames[4];\n    char unused[1];\n    struct {\n        char offset[12];\n        char numbytes[12];\n    } sparse[4];\n    char isextended[1];\n    char realsize[12];\n\n    // 17 remaining bytes of the header are unused.\n    char pad[17];\n};
\n

Observe that, unlike the UStar format, there is no char prefix[155] field. Furthermore, instead of\npax extended headers with a fully general keyword/value format, GNU tar does two things:

\n

  • It goes way harder than UStar on trying to re-use the existing 512-byte V7 member header. In

    particular, GNU tar adds its own base-256 encoding: if the first byte in a numeric field has its\nhigh bit set, then the remaining octets in that field are binary, rather than tar's normal octal\nform.

    \n

    This effectively expands the per-byte entropy in numeric fields from 3 bits to 8 bits. For a\nchar size[12], that makes the effective size limit 2**8014 or one yobibyte.

  • Instead of general x and g extended headers, GNU tar introduces a bunch of specialized

    typeflags for its own extensions.

    \n

    Many of these are not relevant to the post, or even to modern tar archives at all. But two are:

    \n
    • The L typeflag identifies the next sequence of content blocks as containing a filename.
    • The K typeflag identifies the next sequence of content blocks as containing a link name.
    \n

    These two typeflags effectively fill the role of the pax path and linkpath records,\nrespectively.

\n

To visualize that:

\n

For this example, a GNU tar implementation should extract two members:

\n

  • hasdrubal.txt with a payload of 10737418240 bytes.

    Observe that this size, which exceeds 8 GiB, was encoded directly in the header. However, the\neffective filename of hasdrubal.txt comes from overriding the header's original filename of\nhannibal.txt via the preceding GNU L extension.

  • publius.txt, which links to another file named africanus.txt.

    Observe that publius.txt originally declared publius.txt -> scipio.txt as its link, but this\ngets overridden to africanus.txt by the preceding GNU K extension.

\n

Finally, while GNU tar does not have pax's notion of global vs. local precedence, GNU tar extensions\ncan stack. For example, to override a member's name and linkname at the same time, you'd have\na tar header sequence like this15:

\n

tar-codec

\n

State machine turducken

\n

Now that we understand very roughly how the various tar family formats look and work, let's see\nwhat tar-codec has to do to parse them safely and efficiently.

\n

First we'll revisit the state machine. Our nice and simple V7 state machine now understands the\ndifferent pax and GNU states, and has become this monstrosity:

\n

(Note that the error transitions in this state machine are dramatically simplified: in practice, any\nI/O or validation failure at any point also results in a Failed state.)

\n

What in the world happened? A few different things:

\n

  • Our state machine begins with a "classification" stage, which effectively locks the remainder of

    the parse to a single family (either UStar/pax or GNU16) based on the identity (magic and\nversion) of the first block.

    \n

    This is done for differential reduction reasons: while researching tar-codec's design, we\nrealized that many other widely used tar parsers attempt to merge states across various tar\nfamilies, with inconsistent results.

    \n

    For example, with some tar parsers, it's possible to stack headers like so:

    \n

    This has unclear semantics: GNU tar expects an L or K to apply to the next "file" member, but\nthe next physical member in the stream is a pax local extension that also wants to modify the\nnext "file" member. In practice we've seen other parsers do all kinds of things in this state,\nincluding attempting to merge the extensions, honoring one over the other(s), attempting to apply\nthe L state to x instead of the file member, and so forth. Each of these is a potentially\nexploitable differential!

    \n

    Instead, tar-codec will reject a tar archive if its individual headers don't agree about what\nfamily of tar it is. Real-world tar encoders don't mix tar families (there's no good reason to),\nso this doesn't pose a significant breakage risk.

  • The GNU part of the state machine is slightly more strict than the GNU documentation implies: the

    GNU documentation allows stacking of L/K extensions, including multiple of the same kind, in\nany order.

    \n

    tar-codec tightens that to allowing L and K in any order, but only one of each per\nunderlying member.

  • The UStar/pax part of the state machine models local and global extended headers rigidly.

    There's a significant amount of complexity beneath this: pax allows x and g anywhere in the\ntar stream. For example, this is a perfectly valid extension sequence in pax:

    \n

    This brings us to one of the most painful parts of pax that we glossed over before: extended\nheaders don't shadow each other fully, they compose depending on what state the "stack" of\neffective pax records is in. The state of any given pax record can be:

    \n

    • Undeclared: no effective pax record exists anywhere in the current stack of local and global

      extensions, which means that the member's own UStar header's state takes precedence.

    • Active: there's an effective pax record in the stack, and it has a value.

    • Tombstoned: there's an effective pax record in the stack, but it has no value (for example,

      8 path=\\n instead of 12 path=abc\\n).

      \n

      You would think that this would be the same as the Undeclared state, but it's not! pax\nconsiders this record explicitly deleted, and prescribes:

      \n

      If the <value> field is zero length, it shall delete any header block field, previously\nentered extended header value, or global extended header value of the same name.

      \n

      This has fascinating consequences for the state machine, because it means that an\neffective-but-deleted path record means...a file with no path? How are you supposed to extract\nthat17?

    \n

    Needless to say, tar-codec tries to follow this faithfully. In particular, it attempts to be\ndiligent about rejecting logically-consistent-but-nonsense states that fall out of pax's strange\nrecord state machine, including empty paths and global extended headers that try to declare size\nand other key states for all members.

\n

From parsing to extraction

\n

One of the early design choices we made with tar-codec was to isolate the implementation into a\nfew layers. From lowest to highest:

\n

  • The tar-framing crate is responsible for taking a stream of blocks and turning it into a stream

    of frames. This happens in two refinement stages:

    \n

    • The stream or "physical" API honors the big state machine above and yields each block in a given

      state, one block at a time.

    • The "logical" API consumes the blocks produced by the "physical" API, and yields them as a

      stream of assembled members. Each assembled member is essentially its "header" (which is the\nphysical header, merged with any effective extension state that affected the physical framing),\na separate extension state for non-framing extensions (such as GNU L and K), and a readable\npayload for the member's actual contents.

  • The tar-codec crate takes the "logical" API from tar-framing and turns it

    into something that can be extracted in memory or onto disk.

\n

Extracting an archive (of any format) onto disk is its own can of worms, with its own differential\nrisks18. Another one of our early observations is that every archive implementation tends to\nhave its own extract-to-disk implementation, with its own edge cases.

\n

We built archive-trait to address that. archive-trait exposes Archive and ArchiveBuilder\ntraits (along with associated types and machinery) that allow any format to expose uniform APIs for\nreading and creating archives. We then built archive-trait's own filesystem extraction on top of\ncap-std, giving us the ability to "anchor" extraction into a single directory and avoid a large\nclass of link and path traversal footguns.

\n

Lessons learned

\n

  • Writing a differential-resistant tar parser is surprisingly hard, given how ubiquitous tar files

    of all flavors are, and how simple the original (V7) format is.

    \n

    This post has only scratched the surface in terms of complexity. To recap:

    \n

    • Each tar family has its own state machine, which is not always clearly laid out in that family's documentation

      or reference implementation. It is imperative that these state machines do not mix, but real-world tar\nparsers regularly allow them to.

    • Each tar family has its own, non-trivial rules for determining the precedence of state (pathname, linkname, and size)

      attached to archive members.

      \n

      Some of this state affects the framing of the parser itself, meaning that faithfully\nhandling e.g. pax archives means handling size and similar records up-front, rather than trying to abstract the\nparse across individual 512-byte header blocks.

    \n

    Separately, there were numerous places where we had to compromise on strict pax conformance\nbecause of observed behavior from other implementations:

    \n

    • The pax spec is explicit that pax record values must be UTF-8. However, both GNU tar and star historically emitted

      some vendor-specific records19 as raw binary. See this bug-tar thread for more context.

    • UStar and pax are explicit that uname and gname should be NUL-terminated in the UStar header, meaning that each can

      be at most 31 bytes long. However, CPython's tarfile (and probably others) will happily fill either to 32 bytes, only\noverflowing to the pax record at 33 bytes or more. See CPython#159322 for more context.

    • More generally, tar implementations will frequently leave mode, uid, gid, and mtime as all NUL, rather than

      zeroed or similar. This doesn't pose a framing/parser synchronization risk, so tar-codec tracks them as "missing"\nrather than assigning an arbitrary default value.

    \n

    Finally, in the process of designing and implementing tar-codec, we found many parser\ndifferential primitives that cause various tar implementations to behave very differently on\nidentical inputs. Some of these are currently being disclosed as security vulnerabilities.\nHowever, some of the general classes we saw:

    \n

    • pax/GNU state machine confusion, as mentioned above. In particular, some tar implementations can be fooled

      into applying a pax extension's states to a GNU extension (or vice versa) instead of to the subsequent\nmember, often resulting in full desynchronization and different extraction results.

    • Mis-handling of pax extension precedence, in particular both global (g) extended headers and

      record "tombstoning." For example, some implementations will accept a g header with size=1234\nbut ignore it, despite pax saying clearly that such a size record overrides any UStar size fields.\nThis also typically results in a full desynchronization primitive.

    • Mis-handling of pax record parsing. For example, some implementations will ignore the <length>

      prefix and instead assume that pax records can be safely split on \\n. This is not true, since\nthe <value> of a pax record can contain a verbatim \\n. As you may have guessed, this also\ntypically results in a full desynchronization primitive.

    • More generally, we found a great deal of variety in how lax various tar implementations are when it comes to parsing

      integer fields: many use their language's native "integer from string" API, which can behave surprisingly\nin the presence of different (non-Western Arabic) numeral forms, leading and trailing whitespace,\nand so forth. These small variations can percolate into two different implementations parsing\nthe same size field as different integers, once again resulting in desynchronization.

  • Decode and encode performance was not the hard part: we found a lot of low-hanging fruit while

    optimizing both, including optimizing tar's checksum generation and calculating pax record length\nprefixes.

    \n

    We also picked up significant performance gains by ensuring that tar-codec reuses allocations and\nbatches/buffers its work wherever possible (and where doing so does not weaken the API's internal\nconsistency or cancellation correctness in an async setting).

  • Conversely, performant secure extraction is somewhat hard: directory capability tracking (of

    the kind that cap-std does) adds measurable overhead to many of our filesystem operations,\nparticularly when extracting on hosts that lack openat2(2) or an equivalent system\ncall20. This negatively impacts extraction on macOS in particular.

\n

All in all, tar-codec's design rests on well-worn principles for secure parser design: we are\nconservative rather than liberal in what we accept, and we enforce correctness by construction\nthrough increasingly constrictive layers (from member framing to streaming to extraction). These\ndesign choices have made us confident that tar-codec is more resistant to differentials (by\ncorrectly rejecting them) than other tar implementations.

\n

The fractured and under-specified nature of tar makes it impossible for us to say that we are\nentirely devoid of differentials, or that new differentials won't be discovered in the future.\nHowever, we believe that tar-codec's primitives are the correct ones if and when those\ncontingencies occur.

\n

Looking forwards

\n

This is not the end of our work, on several levels:

\n

  • While tar-codec is essentially feature-complete for our own purposes (within uv), we're

    interested in making sure that it also solves problems that the larger Open Source community has\naround secure and performant tar parsing.

    \n

    To that end, we'll be looking at supporting other, more niche aspects of the various tar families\n(like sparse files, and more control over vendor-specific pax records) as the need arises.

    \n

    Separately, we'll be working with others in the community on improving the correctness of other\ntar implementations. In particular, we see Harmen Stoppels's nascent tar-minefield project as a\npromising way to test and re-align other implementations with stricter, more\ndifferential-resistant interpretations of the tar family formats.

  • Separately, as you may have guessed from our decision to build a generic archive-trait library:

    tar is not the only format we're thinking about here!

    \n

    We're also actively looking into replacing uv's ZIP parsing, for similar reasons. We hope to share\nour lessons from that work soon, along with material benefits to uv itself.

\n

Footnotes

\n

  1. tar-codec has been available since 0.12, but was disabled by default behind the

    --preview-features tar-codec flag. As of 0.13, it is the default and can be disabled by\n setting UV_LEGACY_TAR_BACKEND=1. ↩

  2. The term "built" is the source of a great deal of confusion for newcomers when they first

    encounter Python packaging. In the context of Python packaging, "built" refers to the fact that\n the distribution is produced by a "build backend", i.e. contains structured metadata that a\n source distribution is not guaranteed to have.

    \n

    It does not imply that the wheel contains shared libraries or other native code, although it can.\n Pure Python packages are also regularly distributed as built distributions, because installing them is\n faster and more reliable. ↩

  3. Historically, a source distribution was literally just a source archive, with the convention

    of having a pyproject.toml or setup.py at the archive's root. More recent Python packaging\n standards state that source distributions must contain a PKG-INFO file containing standard\n metadata. See PEP 643 for more on that.

    \n

    As a result, many source distributions contain no metadata other than the build system itself, which must be\n executed to access the distribution's metadata. ↩

  4. We recommend Edera's "TARmageddon" blog post for a detailed treatment of one such

    differential. ↩

  5. Martin Dørum has a great blog post on the original tar format and how it evolved over time

    into the constellation of variants that currently exist. We highly recommend reading his post. ↩

  6. Whether or not UStar and pax should be clumped together in the same family is doubtless

    extremely controversial to the dozen-or-so other people on Earth who have thought about the tar\n format this much. I consider them to be in the same family, since pax is effectively a strict\n superset of UStar and comes from the same standards body (POSIX) that produced UStar. ↩

  7. There are actually two different GNU tar formats: there's oldgnu i.e. "old GNU" and gnu i.e.

    "new GNU". The difference between these two is minor (relative to everything else) and GNU tar\n itself emits "new GNU" as of tar 1.12, which was released in 1997. So that's enough about "old\n GNU." ↩

  8. "tar" of course being short for "Tape ARchive." Magnetic tapes (like all mass storage) store

    their contents in sequences of fixed-sized blocks, and tar seals 1979's glorious 512-byte blocks\n in amber.

    \n

    tar is also far from the only block-style file format. cpio is another well-known one. ↩

  9. Archive members are things like files and directories, but also things like sockets and device

    files. I'm not really clear on what it means to archive a socket, but both UStar/pax and GNU tar\n will happily let you express that you've done so! ↩

  10. Before compression, of course. tar knows nothing about compression. It's just the format that

    encodes one or more files into a single file. ↩

  11. V7 tar handles directories in two ways: it creates directories that are implied to exist (for

    example, a member whose name is foo/makeme/bar will create foo and foo/makeme if needed),\n and it will explicitly create a directory if a regular file member ends with a slash, e.g.\n foo/makeme/. ↩

  12. In other words, it's a self-describing length. This is somewhat unusual when compared to a

    "normal" TLV/LTV encoding, where the length component is typically either a fixed length or\n something with only limited length variation, like LEB128. ↩

  13. You might be thinking: why bother putting states in the member's UStar header at all, if they're

    just going to be overridden by the pax extended header? Why not just fill those parts of the\n block with NULs?

    \n

    The answer has to do with backwards compatibility: both GNU and pax do a disgusting hack\n interesting trick where the extension/extended header's own name field is also a valid\n path, conventionally a specially formatted one like %d/PaxHeaders.%p/%f, where %d is\n the dirname of the pax path record, %p is the current PID, and %f is the basename\n of the same path record.

    \n

    The theory behind this is (probably) that a subsequent tool can repair the extraction,\n by looking at the pax headers which have been extracted as a regular file instead.\n In practice I suspect this doesn't work well in the greater-than-8 GiB case, which\n is one of the cases where it matters the most. ↩

  14. This is 2**80 instead of 2**96 because we lose size[0] to the base-256 sign bit and

    size[11] to the NUL byte. Implementations that don't require a NUL terminator for fields like\n size could in principle allow up to 2**88 instead. ↩

  15. The order of GNU extensions before the member they affect does not appear to matter in any

    implementation. In other words, you can have L -> K -> 1 or K -> L -> 1. ↩

  16. That's correct: we don't support parsing V7-style tar files. An early review revealed that

    they're nonexistent in the Python packaging ecosystem, probably because Python's own tarfile\n never supported creating them. The gap between V7 and UStar is small, however, so adding it\n would not be difficult if needed. ↩

  17. Many tar implementations do random things here, like trying to extract to an empty path (which

    of course fails), or picking some previous filename or extended header state. ↩

  18. Extraction differentials themselves are out of this post's scope, but to summarize: extraction

    depends heavily on the target filesystem and its state, including any pre-existing symbolic and\n hard links.

    \n

    Nuances abound, including how to handle files that differ in casing inside the archive but not on the filesystem itself\n (macOS and Windows), how to handle symlinking on systems where symbolic links are privileged (Windows), and how to handle "funky but valid" filenames\n (non-UTF-8, UTF-8 with NULs, control characters, &c.). ↩

  19. A pax "vendor record" is a normal pax record, but with a name like VENDOR.keyword instead of

    keyword. pax itself does not specify their semantics and various implementations do wildly\n different things with them, so tar-codec rejects them by default (with an option to skip/ignore\n them instead). SCHILY.xattr.* is one common example of a vendor record namespace. GNU also has\n their own vendor namespace, e.g. GNU.sparse.* for sparse members. ↩

  20. Without openat2(2) or a similar source of RESOLVE_BENEATH semantics, cap-std needs to open

    each directory component in a path, which means that its system calls scale linearly with the\n depth of the paths it needs to track. This is on top of the system call overhead for the\n extraction I/O itself. ↩

\n
\n

Originally published at Astral.

","headings":[{"level":1,"text":"tar-codec: a fast and secure tar parser for the Rust ecosystem","id":"tar-codec-a-fast-and-secure-tar-parser-for-the-rust-ecosystem"},{"level":2,"text":"Background: uv and tar","id":"background-uv-and-tar"},{"level":3,"text":"Why a new tar parser?","id":"why-a-new-tar-parser"},{"level":2,"text":"(t)archaeology","id":"t-archaeology"},{"level":3,"text":"tar problems","id":"tar-problems"},{"level":3,"text":"UStar and pax","id":"ustar-and-pax"},{"level":3,"text":"GNU tar","id":"gnu-tar"},{"level":2,"text":"tar-codec","id":"tar-codec"},{"level":3,"text":"State machine turducken","id":"state-machine-turducken"},{"level":3,"text":"From parsing to extraction","id":"from-parsing-to-extraction"},{"level":2,"text":"Lessons learned","id":"lessons-learned"},{"level":2,"text":"Looking forwards","id":"looking-forwards"},{"level":2,"text":"Footnotes","id":"footnotes"}]}}