{"article":{"slug":"safe-simd-in-rust-even-on-the-inside","title":"Safe SIMD in Rust, even on the inside","subtitle":null,"summary":"Sergey Davidoff explains why raw SIMD intrinsics force unsafe code in Rust and how the fearless_simd crate removes unsafe even from its own internals, using CPU feature tokens, generics, and aggressive inlining while working around ABI pitfalls.","content_type":"blog_post","language":"en","canonical_url":"https://shnatsel.github.io/safe-simd-in-rust-even-on-the-inside/","author":{"name":"Sergey Davidoff","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Sergey \"Shnatsel\" Davidoff","url":"https://shnatsel.github.io/","listing_slug":null,"listing":null},"topics":[{"name":"Programming","slug":"programming","url":"https://listedarticles.com/topics/programming"},{"name":"Rust","slug":"rust","url":"https://listedarticles.com/topics/rust"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":2175,"reading_minutes":9,"published_at":"2026-06-19T00:00:00.000Z","added_at":"2026-10-11T23:10:04.572Z","updated_at":"2026-10-11T23:10:04.572Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/safe-simd-in-rust-even-on-the-inside","markdown_url":"https://listedarticles.com/articles/safe-simd-in-rust-even-on-the-inside.md","example":false,"citation":"Sergey Davidoff, Sergey \"Shnatsel\" Davidoff. \"Safe SIMD in Rust, even on the inside.\" 19 Jun 2026. https://shnatsel.github.io/safe-simd-in-rust-even-on-the-inside/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://shnatsel.github.io/safe-simd-in-rust-even-on-the-inside/"},"body_markdown":"*Rust’s* [*SIMD abstractions*](https://shnatsel.medium.com/the-state-of-simd-in-rust-in-2025-32c263e5f53d) *were not as safe as I’d like. Until now.*\n\nIt’s no secret that raw SIMD intrinsics are unpleasant to use.\n\nYou want to write `a + b`, not this monstrosity:\n\n```\nunsafe {\n #[cfg(all(any(target_arch = \"x86\", target_arch = \"x86_64\"), target_feature = \"avx2\"))]\n _mm256_add_ps(a, b)\n #[cfg(all(any(target_arch = \"x86\", target_arch = \"x86_64\"), target_feature = \"sse\", not(target_feature = \"avx2\")))]\n _mm_add_ps(a, b)\n #[cfg(all(target_arch = \"aarch64\", target_feature = \"neon\"))]\n vaddq_f32(a, b)\n}\n```\n\nLook at it. It’s hideous. And the whole thing is wrapped in `unsafe`!\n\nAnd that’s a simplified example. It still doesn’t handle:\n\n* Other common platforms: AVX-512, 32-bit ARM, WebAssembly* Platforms without SIMD or obscure platforms like RISC-V* Actually loading data like `&[f32]` into a form that each intrinsic accepts* Selecting the best implementation for the CPU it’s running on\n\nLuckily, Rust provides [many SIMD abstractions](https://shnatsel.medium.com/the-state-of-simd-in-rust-in-2025-32c263e5f53d) that handle all of that for you and let you simply write `a + b`.\n\nThere is just one wrinkle. Inside, they’re still full of `unsafe`. It wasn’t gone, just hidden. Vast quantities of it lurking just beneath the surface, getting [screwed](https://github.com/servo/pathfinder/issues/588) [up](https://github.com/wingertge/macerator/issues/31) [occasionally](https://github.com/sarah-quinones/pulp/issues/28).\n\nOr rather, they were. Until now.\n\n## Why do we even need ‘unsafe’?\n\nFor the longest time you couldn’t get around wrapping the call to each intrinsic function such as `_mm256_add_ps` into `unsafe` because it is illegal to call one when it’s not available on the CPU you’re running on.\n\nSo you *had* to have some mechanism for tracking which instructions are needed for each intrinsic, and which instructions you have access to, and cross-referencing them to decide if it’s safe to call a given function.\n\nIt was either tedious if done by hand or complex if done by a code generator, always error-prone, and required `unsafe` around every intrinsic.\n\nThis changed in Rust 1.87 when the compiler started tracking the required instruction sets itself, so you could write this:\n\n```\n#[target_feature(enable = \"avx2\")]\nfn add_avx2(a: __m256, b: __m256) -> __m256 {\n _mm256_add_ps(a, b) // this is an avx2 intrinsic\n}\n```\n\nLook ma, no `unsafe`!\n\n…yet.\n\nYou still cannot write `a + b` with this. The best you can do is this:\n\n```\nunsafe { add_avx2(a, b) }\n```\n\nThis only shifts the `unsafe` up a layer. You can call intrinsics inside functions annotated with the correct `#[target_feature]` now, but there still has to be `unsafe` somewhere in the chain.\n\nThe other problem is more fundamental. You cannot put `#[target_feature]` on the implementation of `+` for your type, because `+` must be available always. So no `a + b` for us using this mechanism.\n\n## Lemma: CPU feature tokens\n\nTo understand how the final solution works, you first need to understand how CPU feature detection works.\n\nNormally, checking for a CPU feature like AVX2 is done at runtime using `is_x86_feature_detected!(\"avx2\")`. But we definitely don’t want to run this check every single time we add two numbers together — that would completely tank performance. We want to check it *once*, and then prove to the compiler that it’s safe to use AVX2 instructions from that point on.\n\nInstead we can encode this proof into the type system using an *unforgeable token*: a zero-sized type with a private inner field. The *only* way to obtain this token is to call a function that performs the CPU feature check. If the check passes, the function hands you the token:\n\n```\npub struct Avx2(());\n\nfn detect_avx2() -> Option {\n if is_x86_feature_detected!(\"avx2\") {\n Some(Avx2(()))\n } else {\n None\n }\n}\n```\n\nAnd because it’s a zero-sized type, passing this token around has no runtime overhead. It exists purely as a compile-time proof.\n\nThe upshot is that as long as you have an instance of the `Avx2` struct, you can be sure that AVX2 instructions are available on the system.\n\n## The key insight\n\nThe compiler doesn’t know it, but this function is safe to call:\n\n```\n#[target_feature(enable = \"avx2\")]\nfn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n _mm256_add_ps(a, b)\n}\n```\n\nYou can *only* call this function if you have an `Avx2` token, which you can only get if AVX2 instructions are available on the system.\n\nIf we can explain to the compiler that this is valid (using `unsafe`), we can write that `unsafe` only once and reuse it everywhere.\n\nWhat we need is a macro which is safe to invoke:\n\n```\nwith_avx2!(\n fn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n _mm256_add_ps(a, b)\n }\n)\n```\n\nbut expands into this behind the scenes:\n\n```\nfn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n // SAFETY: Avx2 is available according to the token,\n // and we verified that the inner function is not an `unsafe fn`\n unsafe { inner(token, a, b) }\n\n #[target_feature(enable = \"avx2\")]\n fn inner(token: Avx2, a: __m256, b: __m256) -> __m256 {\n _mm256_add_ps(a, b)\n }\n}\n```\n\nNow if you use an intrinsic that isn’t in AVX2, **the compiler will reject it!**\n\nWe’ve just managed to provide a safe programming interface to SIMD intrinsics without any bespoke tracking of target features!\n\nEven though there is one `unsafe` block still inside it, it’s encapsulated in a sound API, so you cannot misuse it to cause memory safety bugs. In that sense it is just like `println!`, safely abstracting unsafe code.\n\nThis way you only ever need to review and audit *this one macro,* not hundreds upon hundreds of bespoke `unsafe` blocks. And the only things we could *possibly* screw up in the implementation are:\n\n1. Mapping the token to the wrong `#[target_feature]`- Allowing an `unsafe fn` to be called from a safe context\n\nAnd both failure modes are quite easy to check for.\n\nSo now we can call `add_avx2(token, a, b)` without `unsafe`, but that still doesn’t get us to `a + b`. How do we solve *that?*\n\n## Generics to the rescue\n\nWe cannot annotate the implementation of `a + b` with `#[target_feature]` because it must be safe to call from anywhere. And we cannot pass a token into the function because it accepts `a` and `b` but not `token`.\n\nBut even if we could do that, it would make for a pretty ugly API. We want `a + b` to always work and automatically use the best SIMD instructions without the user ever fussing with tokens.\n\nWe can use generics to solve both problems at once: by defining an `f32x8` type that’s generic over the available instruction sets, we can implement addition on it that both smuggles a token inside it *and* creates a separate implementation for each SIMD instruction set!\n\nThis is what it looks like:\n\n```\npub trait Level {}\n\n#[derive(Clone, Copy)]\npub struct Avx2(());\nimpl Level for Avx2 {}\n\npub struct f32x8 {\n // For simplicity we'll back this with an array in this example.\n // In production code we use native SIMD types for the level.\n data: [f32; 8],\n // The smuggled token!\n token: L,\n}\n\n/// implementation of `a + b` for Avx2\nimpl std::ops::Add for f32x8 {\n type Output = Self;\n\n fn add(self, rhs: Self) -> Self::Output {\n // (type conversions abbreviated)\n // Use the Avx2 token to call our safe wrapper\n let result = add_avx2(self.token, self, rhs);\n Self {\n data: store_m256(result),\n token: self.token,\n }\n }\n}\n```\n\nAnd then we can just as easily make it work for any other instruction set, or when SIMD is not available at all:\n\n```\n#[derive(Clone, Copy)]\npub struct NoSimd(());\nimpl Level for NoSimd {}\n\n/// implementation of `a + b` when no SIMD is available\nimpl std::ops::Add for f32x8 {\n type Output = Self;\n\n fn add(self, rhs: Self) -> Self::Output {\n let result = std::array::from_fn(|i| self.data[i] + rhs.data[i]);\n Self {\n data: result,\n token: self.token,\n }\n }\n}\n```\n\nWe’ve just solved safety *and* runtime instruction selection at once!\n\nAdd a convenience function that gives you the best `Level` available on the system, and you get pretty much the perfect API for SIMD!\n\n## The ABI would like a word\n\nThere is, unfortunately, a fundamental problem to writing `a + b` and having it lower to SIMD instructions: **function call overhead.**\n\nCalling a function is not free but pretty cheap — just a handful of CPU instructions. But a *handful* of instructions is a lot more than *one* instruction we’ve just used for implementing addition!\n\nSo if there’s a function call in the way, addition performance will plummet. And performance is the entire point of using SIMD in the first place!\n\nThe compiler is usually pretty good at erasing this overhead via [inlining](https://matklad.github.io/2021/07/09/inline-in-rust.html). It basically copy-pastes the implementation of a function you called into the function calling it, so there’s no more function and no more overhead.\n\nBut `#[target_feature]` annotations throw a wrench in the works. The compiler cannot inline a function that has a `#[target_feature]` annotation into one that doesn’t, because the required features are not available in it!\n\nAnd guess what cannot have a `#[target_feature]` annotation? Yeah.\n\nSo how *do* we make `a + b` work with SIMD?\n\n## Inline it, inline it with fire!\n\nWe cannot put `#[target_feature]` on the function that implements `a + b`, but we *can* put it on the function that calls `a + b`!\n\nThen we can use [inlining](https://matklad.github.io/2021/07/09/inline-in-rust.html) to get the implementation of `a + b` copied into the function calling it, and it ends up in a `#[target_feature]` context.\n\nSo the call chain looks like this:\n\n```\n#[target_feature(enable = \"avx2\")]\nfn do_stuff() {\n // TODO: some computation\n c = a + b;\n // TODO: some more computation\n}\n\n// which calls into...\n\n#[inline(always)] // Function body will be copied into the caller\nfn add(self, rhs: Self) -> Self::Output {\n // Use the Avx2 token to call our safe wrapper\n add_avx2(self.token, self, rhs);\n // return statement abbreviated\n}\n\n// which calls into...\n\n#[inline] // Function body will be copied into the caller if feasible\n#[target_feature(enable = \"avx2\")]\nfn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n _mm256_add_ps(a, b)\n}\n```\n\nThis works.\n\nYou can add these annotations in the right places and abstract over the SIMD levels and you [don’t even need macros](https://docs.rs/fearless_simd/latest/fearless_simd/trait.Simd.html#tymethod.vectorize).\n\nThe problem is that whenever you call `a + b` on SIMD types, you have to do it from a function with either `#[inline(always)]` or `#[target_feature]` on it, otherwise the code still compiles but performance plummets.\n\nWanna see for yourself? Open [this example](https://rust.godbolt.org/z/xo1fEfqnT), remove `#[target_feature]` from it and watch the generated assembly turn into absolute horror show.\n\nI’m not sure what can be done about this. The limitation seems quite fundamental for any approach that implements `a + b` with SIMD.\n\nThe [Struct Target Features RFC](https://shnatsel.medium.com/github.com/rust-lang/rfcs/pull/3525) solves this for `add_avx2(token, a, b)` and `add::(token, a, b)` but I don’t see a path to `a + b` just yet.\n\n## Where does that leave us?\n\nDespite the inlining wrinkle inherent to all SIMD code, we’ve managed to provide a remarkably pleasant SIMD abstraction at a staggeringly low, never-before-seen level of unsafe code.\n\nYou can find the production version of these ideas in [`fearless_simd`](https://crates.io/crates/fearless_simd) v0.5, **available now** in a package registry near you! And [here’s a small example](https://github.com/linebender/fearless_simd/blob/d2411665f0726cd6c09bd6fd98af4af18e3c1778/fearless_simd/examples/sigmoid.rs) to see how it all fits together in production.\n\nThe macro used to implement it is also exposed publicly, so you can easily [mix and match](https://github.com/linebender/fearless_simd/blob/e76b820154dc09f41bca8302670424d8ca079219/fearless_simd/examples/srgb.rs) high-level operations like `a + b` and platform-specific intrinsics to take full advantage of the hardware.\n\nThere is more than one `unsafe` block in `fearless_simd` because it also provides the functionality of [`safe_unaligned_simd`](https://crates.io/crates/safe_unaligned_simd) crate, but that too is done at a [significantly lower](https://github.com/okaneco/safe_unaligned_simd/issues/51) amount of unsafe code than the original.\n\nFor me the barrier to using high-level SIMD abstractions was always the sheer amount of `unsafe` they brought. It was scary and hard to justify.\n\nBut now SIMD in Rust can be truly fearless.\n\n## Acknowledgements\n\nI am shocked that I am the first to put this into production (as far as I can tell), because I’m certainly not the first person to think of this.\n\nCPU feature tokens are an old and common idea. The [`pulp`](https://crates.io/crates/pulp) crate has been using them for years, but they relied on handwritten unsafe wrappers around intrinsics, and [occasionally got them wrong](https://github.com/sarah-quinones/pulp/issues/28).\n\nGenerating multiple implementations using generics is also an old idea. It is part of the [original `fearless_simd` concept](https://linebender.org/blog/towards-fearless-simd/) from 8 years ago. The [`simdeez`](https://crates.io/crates/simdeez) crate, which predates it, also [seems to use something similar](https://docs.rs/simdeez/1.0.8/simdeez/).\n\nThe key insight of combining tokens with a single safe wrapper that delegates to `rustc` is not unique to me either. Just in the context of `fearless_simd` crate, Raph Levien has [experimented with it](https://github.com/linebender/fearless_simd/commit/81b6ab4c8dd6f25064f539c49818c45c2f686815), and [Daniel McNab](https://github.com/DJMcNab) created a [more elaborate implementation](https://github.com/linebender/fearless_simd/pull/108) than mine months ago.\n\nDaniel’s approach allows fine-grained tracking of every individual CPU feature, as opposed to a handful of fixed CPU feature levels that `fearless_simd` uses. It is more expressive, but came at the cost of complexity, and his approach never got merged because no other maintainer stepped up to review it. I still hope it will be published as a standalone crate someday.\n\nThanks to Daniel and to [Laurenz Stampfl](https://github.com/LaurenzV) for reviewing all my PRs to `fearless_simd`, they were big and the quick reviews are really appreciated!\n\n*This article was originally published [on my Medium](https://shnatsel.medium.com/safe-simd-in-rust-even-on-the-inside-c6f1ff381828).*","body_html":"

Rust’s SIMD abstractions were not as safe as I’d like. Until now.

\n

It’s no secret that raw SIMD intrinsics are unpleasant to use.

\n

You want to write a + b, not this monstrosity:

\n
unsafe {\n    #[cfg(all(any(target_arch = "x86", target_arch = "x86_64"), target_feature = "avx2"))]\n    _mm256_add_ps(a, b)\n    #[cfg(all(any(target_arch = "x86", target_arch = "x86_64"), target_feature = "sse", not(target_feature = "avx2")))]\n    _mm_add_ps(a, b)\n    #[cfg(all(target_arch = "aarch64", target_feature = "neon"))]\n    vaddq_f32(a, b)\n}
\n

Look at it. It’s hideous. And the whole thing is wrapped in unsafe!

\n

And that’s a simplified example. It still doesn’t handle:

\n
  • Other common platforms: AVX-512, 32-bit ARM, WebAssembly* Platforms without SIMD or obscure platforms like RISC-V* Actually loading data like &[f32] into a form that each intrinsic accepts* Selecting the best implementation for the CPU it’s running on
\n

Luckily, Rust provides many SIMD abstractions that handle all of that for you and let you simply write a + b.

\n

There is just one wrinkle. Inside, they’re still full of unsafe. It wasn’t gone, just hidden. Vast quantities of it lurking just beneath the surface, getting screwed up occasionally.

\n

Or rather, they were. Until now.

\n

Why do we even need ‘unsafe’?

\n

For the longest time you couldn’t get around wrapping the call to each intrinsic function such as _mm256_add_ps into unsafe because it is illegal to call one when it’s not available on the CPU you’re running on.

\n

So you had to have some mechanism for tracking which instructions are needed for each intrinsic, and which instructions you have access to, and cross-referencing them to decide if it’s safe to call a given function.

\n

It was either tedious if done by hand or complex if done by a code generator, always error-prone, and required unsafe around every intrinsic.

\n

This changed in Rust 1.87 when the compiler started tracking the required instruction sets itself, so you could write this:

\n
#[target_feature(enable = "avx2")]\nfn add_avx2(a: __m256, b: __m256) -> __m256 {\n    _mm256_add_ps(a, b) // this is an avx2 intrinsic\n}
\n

Look ma, no unsafe!

\n

…yet.

\n

You still cannot write a + b with this. The best you can do is this:

\n
unsafe { add_avx2(a, b) }
\n

This only shifts the unsafe up a layer. You can call intrinsics inside functions annotated with the correct #[target_feature] now, but there still has to be unsafe somewhere in the chain.

\n

The other problem is more fundamental. You cannot put #[target_feature] on the implementation of + for your type, because + must be available always. So no a + b for us using this mechanism.

\n

Lemma: CPU feature tokens

\n

To understand how the final solution works, you first need to understand how CPU feature detection works.

\n

Normally, checking for a CPU feature like AVX2 is done at runtime using is_x86_feature_detected!("avx2"). But we definitely don’t want to run this check every single time we add two numbers together — that would completely tank performance. We want to check it once, and then prove to the compiler that it’s safe to use AVX2 instructions from that point on.

\n

Instead we can encode this proof into the type system using an unforgeable token: a zero-sized type with a private inner field. The only way to obtain this token is to call a function that performs the CPU feature check. If the check passes, the function hands you the token:

\n
pub struct Avx2(());\n\nfn detect_avx2() -> Option<Avx2> {\n    if is_x86_feature_detected!("avx2") {\n        Some(Avx2(()))\n    } else {\n        None\n    }\n}
\n

And because it’s a zero-sized type, passing this token around has no runtime overhead. It exists purely as a compile-time proof.

\n

The upshot is that as long as you have an instance of the Avx2 struct, you can be sure that AVX2 instructions are available on the system.

\n

The key insight

\n

The compiler doesn’t know it, but this function is safe to call:

\n
#[target_feature(enable = "avx2")]\nfn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n    _mm256_add_ps(a, b)\n}
\n

You can only call this function if you have an Avx2 token, which you can only get if AVX2 instructions are available on the system.

\n

If we can explain to the compiler that this is valid (using unsafe), we can write that unsafe only once and reuse it everywhere.

\n

What we need is a macro which is safe to invoke:

\n
with_avx2!(\n    fn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n        _mm256_add_ps(a, b)\n    }\n)
\n

but expands into this behind the scenes:

\n
fn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n    // SAFETY: Avx2 is available according to the token,\n    // and we verified that the inner function is not an `unsafe fn`\n    unsafe { inner(token, a, b) }\n\n    #[target_feature(enable = "avx2")]\n    fn inner(token: Avx2, a: __m256, b: __m256) -> __m256 {\n        _mm256_add_ps(a, b)\n    }\n}
\n

Now if you use an intrinsic that isn’t in AVX2, the compiler will reject it!

\n

We’ve just managed to provide a safe programming interface to SIMD intrinsics without any bespoke tracking of target features!

\n

Even though there is one unsafe block still inside it, it’s encapsulated in a sound API, so you cannot misuse it to cause memory safety bugs. In that sense it is just like println!, safely abstracting unsafe code.

\n

This way you only ever need to review and audit this one macro, not hundreds upon hundreds of bespoke unsafe blocks. And the only things we could possibly screw up in the implementation are:

\n
  1. Mapping the token to the wrong #[target_feature]- Allowing an unsafe fn to be called from a safe context
\n

And both failure modes are quite easy to check for.

\n

So now we can call add_avx2(token, a, b) without unsafe, but that still doesn’t get us to a + b. How do we solve that?

\n

Generics to the rescue

\n

We cannot annotate the implementation of a + b with #[target_feature] because it must be safe to call from anywhere. And we cannot pass a token into the function because it accepts a and b but not token.

\n

But even if we could do that, it would make for a pretty ugly API. We want a + b to always work and automatically use the best SIMD instructions without the user ever fussing with tokens.

\n

We can use generics to solve both problems at once: by defining an f32x8 type that’s generic over the available instruction sets, we can implement addition on it that both smuggles a token inside it and creates a separate implementation for each SIMD instruction set!

\n

This is what it looks like:

\n
pub trait Level {}\n\n#[derive(Clone, Copy)]\npub struct Avx2(());\nimpl Level for Avx2 {}\n\npub struct f32x8<L: Level> {\n    // For simplicity we'll back this with an array in this example.\n    // In production code we use native SIMD types for the level.\n    data: [f32; 8],\n    // The smuggled token!\n    token: L,\n}\n\n/// implementation of `a + b` for Avx2\nimpl std::ops::Add for f32x8<Avx2> {\n    type Output = Self;\n\n    fn add(self, rhs: Self) -> Self::Output {\n        // (type conversions abbreviated)\n        // Use the Avx2 token to call our safe wrapper\n        let result = add_avx2(self.token, self, rhs);\n        Self {\n            data: store_m256(result),\n            token: self.token,\n        }\n    }\n}
\n

And then we can just as easily make it work for any other instruction set, or when SIMD is not available at all:

\n
#[derive(Clone, Copy)]\npub struct NoSimd(());\nimpl Level for NoSimd {}\n\n/// implementation of `a + b` when no SIMD is available\nimpl std::ops::Add for f32x8<NoSimd> {\n    type Output = Self;\n\n    fn add(self, rhs: Self) -> Self::Output {\n        let result = std::array::from_fn(|i| self.data[i] + rhs.data[i]);\n        Self {\n            data: result,\n            token: self.token,\n        }\n    }\n}
\n

We’ve just solved safety and runtime instruction selection at once!

\n

Add a convenience function that gives you the best Level available on the system, and you get pretty much the perfect API for SIMD!

\n

The ABI would like a word

\n

There is, unfortunately, a fundamental problem to writing a + b and having it lower to SIMD instructions: function call overhead.

\n

Calling a function is not free but pretty cheap — just a handful of CPU instructions. But a handful of instructions is a lot more than one instruction we’ve just used for implementing addition!

\n

So if there’s a function call in the way, addition performance will plummet. And performance is the entire point of using SIMD in the first place!

\n

The compiler is usually pretty good at erasing this overhead via inlining. It basically copy-pastes the implementation of a function you called into the function calling it, so there’s no more function and no more overhead.

\n

But #[target_feature] annotations throw a wrench in the works. The compiler cannot inline a function that has a #[target_feature] annotation into one that doesn’t, because the required features are not available in it!

\n

And guess what cannot have a #[target_feature] annotation? Yeah.

\n

So how do we make a + b work with SIMD?

\n

Inline it, inline it with fire!

\n

We cannot put #[target_feature] on the function that implements a + b, but we can put it on the function that calls a + b!

\n

Then we can use inlining to get the implementation of a + b copied into the function calling it, and it ends up in a #[target_feature] context.

\n

So the call chain looks like this:

\n
#[target_feature(enable = "avx2")]\nfn do_stuff() {\n    // TODO: some computation\n    c = a + b;\n    // TODO: some more computation\n}\n\n// which calls into...\n\n#[inline(always)] // Function body will be copied into the caller\nfn add(self, rhs: Self) -> Self::Output {\n    // Use the Avx2 token to call our safe wrapper\n    add_avx2(self.token, self, rhs);\n    // return statement abbreviated\n}\n\n// which calls into...\n\n#[inline] // Function body will be copied into the caller if feasible\n#[target_feature(enable = "avx2")]\nfn add_avx2(token: Avx2, a: __m256, b: __m256) -> __m256 {\n    _mm256_add_ps(a, b)\n}
\n

This works.

\n

You can add these annotations in the right places and abstract over the SIMD levels and you don’t even need macros.

\n

The problem is that whenever you call a + b on SIMD types, you have to do it from a function with either #[inline(always)] or #[target_feature] on it, otherwise the code still compiles but performance plummets.

\n

Wanna see for yourself? Open this example, remove #[target_feature] from it and watch the generated assembly turn into absolute horror show.

\n

I’m not sure what can be done about this. The limitation seems quite fundamental for any approach that implements a + b with SIMD.

\n

The Struct Target Features RFC solves this for add_avx2(token, a, b) and add::<S: Simd>(token, a, b) but I don’t see a path to a + b just yet.

\n

Where does that leave us?

\n

Despite the inlining wrinkle inherent to all SIMD code, we’ve managed to provide a remarkably pleasant SIMD abstraction at a staggeringly low, never-before-seen level of unsafe code.

\n

You can find the production version of these ideas in fearless_simd v0.5, available now in a package registry near you! And here’s a small example to see how it all fits together in production.

\n

The macro used to implement it is also exposed publicly, so you can easily mix and match high-level operations like a + b and platform-specific intrinsics to take full advantage of the hardware.

\n

There is more than one unsafe block in fearless_simd because it also provides the functionality of safe_unaligned_simd crate, but that too is done at a significantly lower amount of unsafe code than the original.

\n

For me the barrier to using high-level SIMD abstractions was always the sheer amount of unsafe they brought. It was scary and hard to justify.

\n

But now SIMD in Rust can be truly fearless.

\n

Acknowledgements

\n

I am shocked that I am the first to put this into production (as far as I can tell), because I’m certainly not the first person to think of this.

\n

CPU feature tokens are an old and common idea. The pulp crate has been using them for years, but they relied on handwritten unsafe wrappers around intrinsics, and occasionally got them wrong.

\n

Generating multiple implementations using generics is also an old idea. It is part of the original fearless_simd concept from 8 years ago. The simdeez crate, which predates it, also seems to use something similar.

\n

The key insight of combining tokens with a single safe wrapper that delegates to rustc is not unique to me either. Just in the context of fearless_simd crate, Raph Levien has experimented with it, and Daniel McNab created a more elaborate implementation than mine months ago.

\n

Daniel’s approach allows fine-grained tracking of every individual CPU feature, as opposed to a handful of fixed CPU feature levels that fearless_simd uses. It is more expressive, but came at the cost of complexity, and his approach never got merged because no other maintainer stepped up to review it. I still hope it will be published as a standalone crate someday.

\n

Thanks to Daniel and to Laurenz Stampfl for reviewing all my PRs to fearless_simd, they were big and the quick reviews are really appreciated!

\n

This article was originally published on my Medium.

","headings":[{"level":2,"text":"Why do we even need ‘unsafe’?","id":"why-do-we-even-need-unsafe"},{"level":2,"text":"Lemma: CPU feature tokens","id":"lemma-cpu-feature-tokens"},{"level":2,"text":"The key insight","id":"the-key-insight"},{"level":2,"text":"Generics to the rescue","id":"generics-to-the-rescue"},{"level":2,"text":"The ABI would like a word","id":"the-abi-would-like-a-word"},{"level":2,"text":"Inline it, inline it with fire!","id":"inline-it-inline-it-with-fire"},{"level":2,"text":"Where does that leave us?","id":"where-does-that-leave-us"},{"level":2,"text":"Acknowledgements","id":"acknowledgements"}]}}