{"article":{"slug":"virtual-oxide-labs-emulating-a-full-oxide-rack","title":"Virtual Oxide Labs: Emulating a Full Oxide Rack","subtitle":null,"summary":"Oxide's Product Assurance team introduces Voxel, a tool that emulates a full Oxide rack, including its real control plane, switches, service processors and customer-like networking topologies, so the team can test current software without scarce hardware; it covers image creation, launch, SP emulation and the MPL 2.0 release.","content_type":"blog_post","language":"en","canonical_url":"https://oxide.computer/blog/voxel-labs","author":{"name":"Steve Karam","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Oxide Computer Company","url":"https://oxide.computer/","listing_slug":"oxide-computer","listing":{"slug":"oxide-computer","name":"Oxide Computer Company","listing_type":"company","url":"https://listedstartups.com/companies/oxide-computer"}},"topics":[{"name":"Infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"},{"name":"Hardware","slug":"hardware","url":"https://listedarticles.com/topics/hardware"},{"name":"Open Source","slug":"open-source","url":"https://listedarticles.com/topics/open-source"},{"name":"Software Engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":4170,"reading_minutes":18,"published_at":"2026-10-14T16:00:00.000Z","added_at":"2026-10-14T23:09:49.783Z","updated_at":"2026-10-14T23:09:49.783Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/virtual-oxide-labs-emulating-a-full-oxide-rack","markdown_url":"https://listedarticles.com/articles/virtual-oxide-labs-emulating-a-full-oxide-rack.md","example":false,"citation":"Steve Karam, Oxide Computer Company. \"Virtual Oxide Labs: Emulating a Full Oxide Rack.\" 14 Oct 2026. https://oxide.computer/blog/voxel-labs (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://oxide.computer/blog/voxel-labs"},"body_markdown":"The trouble with testing and developing for rack-scale hardware is that rack-scale is not all that easy to come by. I mean, nevermind the whole DDR5-as-a-currency thing; even if all the parts were available in abundance, any completed racks go swiftly to the front lines (AKA our customers' data centers).\n\nAs a member of the Product Assurance Team at Oxide, I find that totally unfair yet completely reasonable. And so we have to get a bit creative. We have to be able to:\n\n- Replicate diverse customer networking topologies\n- Run our actual control plane\n- Test up to date versions of our software\n- Represent the hardware components (we make racks, after all) including our switches and microcontrollers\n\nAnd the cherry on top: we need to do that without cutting special \"test version\" releases. Because we want to keep our CI sleek, but also because that wouldn’t be playing fair.\n\nNow, don’t get me wrong — we’re not without our tricks. When I joined Oxide last year,\none of the first things I was introduced to was our *racklettes*; that is, smaller\nthan average racks of four sleds and two switches each. These racklettes can be\ninitialized and run just like a real rack, precisely because they *are* a real rack.\nJust not in any form factor it would be economical to sell. But for testing they\nwork like a charm. We have four of them today, soon to be eight, where we can test\nany combination of hardware/software requirement.\n\nWe also have our various simulated methods of running Omicron. You can read more about\nthose at [How to Run Simulated](https://github.com/oxidecomputer/omicron/blob/main/docs/how-to-run-simulated.adoc)\nand the full(er) [How to Run Guide](https://github.com/oxidecomputer/omicron/blob/main/docs/how-to-run.adoc). I\nwon’t go into a ton of detail here, but suffice it to say that most parts of our control plane (and more) can be run independently, in tandem, and in various degrees of simulation.\n\n## What’s a Voxel?\n\nDo you remember the game Q*bert? It was one of the first wildly popular isometric games, and looked a little like this:\n\nWhy am I showing you a ridiculously small game screenshot? Well, each cube in that image represents a 'voxel,' or a point in 3D space. It’s actually the 3D version of a pixel. Volume Element + Pixel = Voxel. Neat, right?\n\nAnyway, for Oxide, Virtual OXidE Labs (or Voxel) means an incredible amount of fidelity packed into a single piece of hardware. It is the spiritual successor to our prior virtual labs (known as a4x2) which allowed us to hack out decent racks running in VMs.\n\nPay attention to that word \"fidelity\" by the way.\nFor it is, as Bryan Cantrill puts it, [the Prime Directive](https://oxide-and-friends.transistor.fm/episodes/emulating-oxide-how-faux-can-you-go): labs should use\nstock Oxide software without hack or modification.\n\nThis is what Voxel gives us. A complete software emulation stack, built on illumos in Rust, where we can build and test a variety of realistic configurations:\n\n- Sled virtualization provided by [Propolis](https://github.com/oxidecomputer/propolis)\n- BGP (numbered and unnumbered) or static networking with Debian and [FRR](https://frrouting.org/)\n- P4-powered switches via Oxide’s own [SoftNPU, a software P4 switch](https://netstack.fm/episodes/oxide-networking-with-ryan-goodfellow)\n- Microcontroller simulation and emulation for our Service Processor (SP) and Root of Trust (RoT)\n\nAnd what’s more, it gives us all these features in a fairly small footprint. At home,\nI run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and\n96GB RAM. On that computer, I can spin up 6 sleds with ease, either in the form of\none rack or in a multirack configuration (two separate racks, one management plane). All\nit requires is [Oxide Helios](https://github.com/oxidecomputer/helios) and a decently\nfast NVMe drive. Launch, and in about 10 minutes you have a full Oxide rack with each \"sled\"\nin its own Propolis VM — alongside switches (also in Propolis) and other infrastructure.\n\n### Image Creation\n\nRemember how I said we have to use stock Oxide software? That actually makes things easier. The first thing we do (following a common practice in hypervisor tooling) is to create a reusable snapshot.\n\nOxide sleds are only ever one of two things: a normal\nsled (a \"gimlet\" for the first generation, \"cosmo\" for the current) or a \"scrimlet,\"\nwhich is a sled wired to the sidecar switches that runs a switch zone. It’s all\namazingly elegant in its complexity, you should read more about it in\n[our networking docs](https://docs.oxide.computer/guides/architecture/networking).\n\nTo create a snapshot for Voxel, all we must do is create a \"sled\" (virtual machine), install our control plane (Omicron) on it, snapshot it before the point of configuration, et voilà: a ready-to-launch sled. The control plane can be installed one of two ways:\n\n- From source\n- Voxel clones the [Omicron repo](https://github.com/oxidecomputer/omicron/) on GitHub (or uses a locally defined source for rapid development) and uses`cargo` to compile\n- From TUF repo\n- We use [Tufaceous](https://github.com/oxidecomputer/tufaceous) to package Omicron releases using The Update Framework (TUF) and our CI tool[Buildomat](https://github.com/oxidecomputer/buildomat/)\n\nUsing the second option as an example, here’s the beginning of the process where `voxel image create` pulls out the key pieces that make up the sled config:\n\n```\n$ pfexec ./target/debug/voxel image create --from-tuf ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip 5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c\n[voxel] TUF source ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip: system version 23.0.0-0.ci+git5d176e0be7e, omicron 5d176e0be7e\n[voxel] extracted 12 zones from ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip\n[voxel] extracted 3 measurement corpus artifacts\n[voxel] extracting host artifacts -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-host\n[voxel] host boot image 1258295296 bytes, phase 1 rom 33554432 bytes\n[voxel] extracted SP/RoT firmware -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-fw/ca9184058bd7\n```\nAfter a decent amount of unpacking and staging, it will launch a VM that everything gets written/installed to. And finally:\n\n```\n[voxel] ready: voxel-cp version=unknown unpacked_artifacts=18 built=2026-09-06T21:37:58\n[voxel] clearing device-instance map, halting\n[voxel] flushing, up to 25s\n[voxel] stopping hypervisor\n[voxel] writing voxel image voxel-cp-5d176e0be7e-tuf\n```\nThat image is a zfs snapshot which can be used and reused to create our rack.\n\n### Ready for Launch\n\nLaunching the rack uses a similar process, except the control plane is already installed. All we need at this point is:\n\n1. An \"frr\" image (the Debian VMs for networking)\n2. A control plane image (like the one we just created)\n3. The `voxel-init` agent that gives each virtual sled its assignment\n\nThe log output is pretty long, but you can scroll through it if you’d like. I’ll recap after.\n\n```\n$ pfexec ./target/debug/voxel launch --emu\n[voxel] sled-agent config schema (image properties): data_links=Tagged disks=Hardcoded\nWriting key pair to \"./cargo-bay/sprockets/test-root-a.key.pem\"\nWriting key pair to \"./cargo-bay/sprockets/test-signer-a1.key.pem\"\n...\nWriting certificate to \"./cargo-bay/sprockets/test-root-a.cert.pem\"\nWriting certificate to \"./cargo-bay/sprockets/test-signer-a1.cert.pem\"\n...\nWriting pki path to \"./cargo-bay/sprockets/test-sprockets-auth-0.certlist.pem\"\nWriting pki path to \"./cargo-bay/sprockets/test-alias-0.certlist.pem\"\n...\nadd net fdb0:a840:2500:1::/64: gateway fdb0:a840:25ff:0::2\nadd net fdb0:a840:2500:5::/64: gateway fdb0:a840:25ff:0::4\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300\n[identity] seed = 0x2aac7ae3f8cfbb1e (3074967765137734430)\n[identity] SP UID 13120858064df80da1642f2b, RoT UUID 00a94a8e206f3f138a790fba1b12db36\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/sidecar.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/archives/sp.zip\nflashed 786688 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310\n[identity] seed = 0xa65abed0e20d08d1 (11987103162554976465)\n[identity] SP UID 72a3c3327682e0df38de478a, RoT UUID fcb600f7322f3393899f72036aa2b4fd\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320\n[identity] seed = 0xbab29f9c81fda354 (13452990531453690708)\n[identity] SP UID 0883f59f877ab40da88e8072, RoT UUID 15468fb5bffd30d3b381e8759621fb4f\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330\n[identity] seed = 0xea954465eef85936 (16903491980996663606)\n[identity] SP UID 49c0e331f4877c7d0afc71eb, RoT UUID 03ec945840463823a95d38dda9e261d1\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/sp-flash.bin\n[voxel] rack 0 SP fleet up on fdb0:a840:25ff:0::1 (4 SP(s): [33300, 33310, 33320, 33330])\nSep 06 22:15:47.126 INFO launching runner: deployment(voxel) persistent(true) custom_propolis_binary(Some(\"/export/home/helios/propolis-server\")) dataset(testbed/falcon) falcon_dir(.falcon)\nSep 06 22:15:47.126 INFO starting preflight for deployment voxel\nSep 06 22:15:48.162 INFO creating links\nSep 06 22:15:48.162 INFO creating simnet link 'voxel_ce_vn_sim0'\nSep 06 22:15:48.164 INFO creating vnic link 'voxel_ce_vn_vnic0'\n...\nSep 06 22:15:48.249 INFO creating external links\nSep 06 22:15:48.249 INFO creating external link voxel_ce_vn_vnic2\nSep 06 22:15:48.251 INFO creating external link voxel_g0_vn_vnic3\n...\nSep 06 22:15:48.255 INFO creating nodes\nSep 06 22:15:48.255 INFO g0: launching node\nSep 06 22:15:48.257 INFO g1: launching node\n...\nSep 06 22:15:48.290 INFO launched instance g0 with pid 13041 on port 43736\nSep 06 22:15:48.291 INFO g0: instance ensure\nSep 06 22:15:48.295 INFO launched instance g1 with pid 13042 on port 33187\nSep 06 22:15:48.295 INFO g1: instance ensure\nSep 06 22:15:48.297 INFO launched instance g2 with pid 13043 on port 51110\n...\nSep 06 22:15:52.423 INFO g1: instance ensure completed after 0 retries\nSep 06 22:15:52.423 INFO g1: instance run\nSep 06 22:15:52.479 INFO g2: instance ensure completed after 0 retries\nSep 06 22:15:52.479 INFO g2: instance run\nSep 06 22:15:52.534 INFO g0: instance ensure completed after 0 retries\nSep 06 22:15:52.534 INFO g0: instance run\nSep 06 22:16:03.961 INFO cr1: mounting /opt/cargo-bay\nSep 06 22:16:03.969 INFO cr2: mounting /opt/cargo-bay\nSep 06 22:16:03.994 INFO cr1: finished mounting /opt/cargo-bay\nSep 06 22:16:03.995 INFO ce: mounting /opt/cargo-bay\nSep 06 22:16:04.002 INFO cr2: finished mounting /opt/cargo-bay\n...\nSep 06 22:16:09.236 INFO ce: launch start\nSep 06 22:16:09.237 INFO cr1: launch start\nSep 06 22:16:09.237 INFO cr2: launch start\nSep 06 22:16:12.893 INFO ce: [voxel-init] pinned enp0s10 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.893 INFO ce: [voxel-init] NAT rack egress via enp0s10\nSep 06 22:16:12.894 INFO ce: [voxel-init] static edge IP 192.168.68.170/22 added on enp0s10\nSep 06 22:16:12.894 INFO ce: [voxel-init] router bring-up complete\nSep 06 22:16:12.894 INFO ce: launch ok\nSep 06 22:16:12.956 INFO cr1: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.956 INFO cr1: [voxel-init] NAT rack egress via enp0s11\nSep 06 22:16:12.956 INFO cr1: [voxel-init] router bring-up complete\nSep 06 22:16:12.956 INFO cr1: launch ok\nSep 06 22:16:12.985 INFO cr2: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.985 INFO cr2: [voxel-init] NAT rack egress via enp0s11\nSep 06 22:16:12.985 INFO cr2: [voxel-init] router bring-up complete\nSep 06 22:16:12.985 INFO cr2: launch ok\nSep 06 22:16:12.985 INFO g0: launch start\nSep 06 22:16:12.986 INFO g1: launch start\nSep 06 22:16:12.987 INFO g2: launch start\nSep 06 22:16:28.581 INFO g1: [voxel-init] underlay(jumbo)=[\"vioif0\", \"vioif1\"] ext-candidates=[\"vioif2\"]\nSep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0 (2FAKE001-M20): M.2 layout written\nSep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:28.581 INFO g1: [voxel-init] c3t1d0 (2FAKE001-M21): M.2 layout written\nSep 06 22:16:28.582 INFO g1: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:28.582 INFO g1: [voxel-init] c4t1d0 (2FAKE001-U20): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c5t1d0 (2FAKE001-U21): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c6t1d0 (2FAKE001-U22): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c7t1d0 (2FAKE001-U23): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c8t1d0 (2FAKE001-U24): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_626cc492-5a93-4b5c-833b-c45a504d8300)\nSep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_711d472d-edbc-4426-9e33-4b000914f4da)\nSep 06 22:16:28.582 INFO g1: [voxel-init] sled-agent activated\nSep 06 22:16:28.582 INFO g1: [voxel-init] gimlet bring-up complete\nSep 06 22:16:28.582 INFO g1: launch ok\nSep 06 22:16:33.159 INFO g0: [voxel-init] underlay(jumbo)=[\"vioif1\", \"vioif2\"] ext-candidates=[\"vioif0\", \"vioif3\"]\nSep 06 22:16:33.159 INFO g0: [voxel-init] SP network fdb0:a840:25ff:0::2/64 on vioif3 (deprecated as a source)\nSep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0 (2FAKE000-M20): M.2 layout written\nSep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0 (2FAKE000-M21): M.2 layout written\nSep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:33.159 INFO g0: [voxel-init] c4t1d0 (2FAKE000-U20): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c5t1d0 (2FAKE000-U21): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c6t1d0 (2FAKE000-U22): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c7t1d0 (2FAKE000-U23): labeled\nSep 06 22:16:33.160 INFO g0: [voxel-init] c8t1d0 (2FAKE000-U24): labeled\nSep 06 22:16:33.160 INFO g0: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_ab4a1425-a7f2-47c7-9ef8-c70180ef02c9)\nSep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_410f70f9-1eea-49fe-8aae-8331cebc4ea1)\nSep 06 22:16:33.160 INFO g0: [voxel-init] switch0 enforcer started (pid 769), log /tmp/switch-enforcer.log\nSep 06 22:16:33.160 INFO g0: [voxel-init] sled-agent activated\nSep 06 22:16:33.160 INFO g0: [voxel-init] gimlet bring-up complete\nSep 06 22:16:33.160 INFO g0: launch ok\nSep 06 22:16:38.179 INFO g2: [voxel-init] underlay(jumbo)=[\"vioif1\", \"vioif2\"] ext-candidates=[\"vioif0\", \"vioif3\"]\nSep 06 22:16:38.179 INFO g2: [voxel-init] SP network fdb0:a840:25ff:0::4/64 on vioif3 (deprecated as a source)\nSep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0 (2FAKE002-M20): M.2 layout written\nSep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0 (2FAKE002-M21): M.2 layout written\nSep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:38.179 INFO g2: [voxel-init] c4t1d0 (2FAKE002-U20): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c5t1d0 (2FAKE002-U21): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c6t1d0 (2FAKE002-U22): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c7t1d0 (2FAKE002-U23): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c8t1d0 (2FAKE002-U24): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_270f44df-c4df-43fd-b02f-e5440857b547)\nSep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_1c986103-d332-4270-b7e1-9341310a621d)\nSep 06 22:16:38.179 INFO g2: [voxel-init] switch1 enforcer started (pid 779), log /tmp/switch-enforcer.log\nSep 06 22:16:38.179 INFO g2: [voxel-init] sled-agent activated\nSep 06 22:16:38.179 INFO g2: [voxel-init] gimlet bring-up complete\nSep 06 22:16:38.179 INFO g2: launch ok\nSep 06 22:16:38.652 INFO rack-init: waiting for the commission API: switch-zone bootstrap address: no bootstrap address on oxz_switch\nSep 06 22:17:54.860 INFO rack-init: waiting for the commission API: tunnel opened\nSep 06 22:17:59.876 INFO rack-init: commission API reachable on g0\nSep 06 22:18:23.892 INFO rack-init: 3 bootstrap sleds ready\n..+.......+.....+.+..+.......+..+...............+....+.....+.+.....+......+....+...+..+....+.....+++++++++++++++++++++++++++++++++++++++*.....+.+.....+......+.+++++++++++\n............+..+.+..+...+.........+......+...+..........+++++++++++++++++++++++++++++++++++++++*...+...+..+++++++++++++++++++++++++++++++++++++++*..................+.....\n-----\nSep 06 22:18:24.023 INFO rack-init: commission-driven RSS started\nSep 06 22:18:24.027 INFO rack-init: watching RSS progress on the RSS node ...\nSep 06 22:18:24.228 INFO rack-init: polling RSS status via ssh root@192.168.68.167\nSep 06 22:18:32.410 INFO rack-init [9/16]: initializing internal DNS\nSep 06 22:18:49.186 INFO rack-init [11/16]: initializing NTP\nSep 06 22:19:13.745 INFO rack-init [12/16]: waiting for time sync\nSep 06 22:20:51.850 INFO rack-init [12/16]: waiting for time sync ... still working (1m in this step)\nSep 06 22:22:22.134 INFO rack-init [13/16]: waiting for database\nSep 06 22:22:47.334 INFO rack-init [14/16]: initializing cluster\nSep 06 22:22:55.805 INFO rack-init [15/16]: initializing zones\nSep 06 22:24:33.717 INFO rack-init [16/16]: handing off to Nexus\nSep 06 22:26:11.824 INFO rack-init [16/16]: handing off to Nexus ... still working (1m in this step)\nSep 06 22:27:49.894 INFO rack-init [16/16]: handing off to Nexus ... still working (3m in this step)\nSep 06 22:28:06.231 INFO rack-init: complete, rack initialized (RSS run b20a2f8d-e511-4a67-a666-7b74b2ce68d7)\nSep 06 22:28:06.231 INFO launch complete\nSep 06 22:28:06.250 INFO external route set: 198.51.100.0/24 -> 192.168.68.170 (ce)\nSep 06 22:28:06.292 INFO rack: external network reachable (dns 198.51.100.20)\n```\nLike I said, a lot of steps. Here’s what happened:\n\n1. Trust material is generated in the form of [sprockets PKI](https://rfd.shared.oxide.computer/rfd/0238)\n2. Emulated STM32H753 and LPC55 microcontrollers are flashed with [Hubris firmware](https://github.com/oxidecomputer/hubris)\n3. Our [automated network topology suite](https://github.com/oxidecomputer/falcon) (known as falcon) wires the topology with simnet and VNIC pairs for every link, and stages the sleds (g0, g1, and g2 in this case)\n4. Propolis VMs launch; 6 total in this example, with 3 driving the network and 3 as Oxide sleds\n5. `voxel-init` runs: a lightweight Rust agent on each node that\n - Sets up NAT and uplink addressing\n - Configures underlay setup on each sled\n - Writes the M.2 partition layout on emulated boot disks and labels five U.2 data disks\n - Seeds the host phase 2 image and install dataset to the M.2s\n - Assigns the scrimlets and runs a switch enforcer so they \"remember\" their identity across restarts\n - Activates `sled-agent` , the per-sled configuration layer for the control plane\n6. Connects to the commission API for `wicketd` , which brings the rack through its initialization phases\n\nIt’s a complex dance of steps, and I really buried the lede on the whole \"emulated\nmicrocontrollers\" front. But suffice it to say these are the complete and right steps for\nany Oxide rack, except the hardware is represented as distinct pieces of code. For example,\nneed a new interconnect between two virtual racks? Add a couple lines of code in `rack.rs`.\n\nThere are a *few* things that don’t follow the Prime Directive (yet!). For example, those certs\nthat make up the sprockets PKI up top aren’t set at manufacturing and coming from the RoT\nlike our production racks. But that’s one of those things that we can live with for now, as\nit doesn’t affect actual rack runtime.\n\n## Emulating Service Processors\n\nThis is the bit I mentioned a moment ago. A big part of what makes our racks unique is\nthe embedded [RoT and SP](https://docs.oxide.computer/guides/architecture/rack-components#_service_processors)\nin every sled. We’ve long had an `SpSim` construct that served as a stub for the service\nprocessor and would return the right info at the right times. But that’s simulation, not\nemulation. It’s always going to be happy path, and that’s just not enough fidelity for\nreal testing cases.\n\nAs such, we built actual emulated Cortex-M7 and Cortex-M33 devices for illumos and Linux\n(x86 and ARM). The [repo for this tool (sp-emu)](https://github.com/oxidecomputer/sp-emu)\nis open source. Each of the two devices runs in a single thread, and (mostly)\nfaithfully emulates the three main requirements: instructions, time, and interrupts.\n\nFrom there, you can flash our Hubris firmware directly to these emulated devices, and then\ncommunicate by way of an ordinary socket. The Voxel lab can speak with these virtual devices via the\nManagement Gateway Service (MGS) and Inter-Processor Communication Channel (IPCC). You can learn\neverything you want and more about these concepts in the [Management Gateway Service repo](https://github.com/oxidecomputer/management-gateway-service/) and [Hubris docs](https://hubris.oxide.computer/reference/).\n\nThere really is no \"run real microcontrollers as virtual devices with this one neat trick!\"\nI can think of. But the real challenge of sp-emu’s job is time. Remember all of the code\nwe’re running there is meant for silicon. Hubris schedules off a 1ms SysTick and the emulator doesn’t\nuse wall time; as such, we had to invent a clock. That said, emulated hardware is also\n*slower* than the real thing. So there’s… well, less time per time. So instead of trying\nto get all temporal about it, time became instructions. For example, I give you `maybe_tick`:\n\n```\nif self.systick == 0 {\n self.systick = self.syst_rvr;\n}\nself.systick -= 1;\nif self.systick == 0 {\n self.systick = self.syst_rvr;\n self.tick_events = self.tick_events.wrapping_add(1);\n if self.syst_csr & 2 != 0\n && self.mode == Mode::Thread\n && !self.primask\n {\n self.exception_entry(15, bus); // SysTick exception\n }\n}\n```\nIn this case, we count instructions against the kernel’s own reload value and fire exceptions anywhere the single-priority design makes it safe to take it — in thread mode, interrupts unmasked. By the same token, peripheral access costs one tick of our clock (an instruction), whereas on silicon it might stall for dozens of cycles on the bus.\n\nYou may have noticed `self.exception_entry` which is defined\nelsewhere in the code:\n\n`fn exception_entry(&mut self, vecnum: u32, bus: &mut Bus)`\n\nIn short, this function models everything the hardware does on a Cortex-M core taking\nan exception, all the way between \"exception becomes active\" to the first handler\ninstruction. `vecnum` here is an ARMv7-M exception number (also the vector table index\nfor that exception’s handler). `bus` is the memory system which is used to actually\npush the frame and read the table.\n\nHubris has a non-preemptible kernel, though it schedules tasks preemptively. SVCall,\nPendSV, SysTick and all IRQs run at the same\npriority level, while faults keep a higher priority. As such, the emulator doesn’t have\nto nest handlers. It processes pending IRQs based on the PRIMASK, FAULTMASK, and BASEPRI\nsettings. If one of them pended PendSV, that will perform the deferred context\nswitch for the kernel. For more details on how it works internally, you can check\n[this lovely bit of documentation](https://github.com/oxidecomputer/hubris/blob/master/sys/kern/src/arch/arm_m.rs#L5-L71).\n\n```\nif let Some(irq) = bus.next_irq() {\n if self.basepri == 0 || (bus.irq_prio(irq) as u32) < self.basepri {\n bus.clear_pending(irq);\n self.exception_entry(16 + irq as u32, bus);\n return;\n }\n}\nif bus.take_pendsv() {\n self.exception_entry(14, bus);\n}\n```\nThere’s a lot more we can talk about here, but to sum it all up you can look at this pseudo-ish loop:\n\n```\nloop {\n cpu.step(&mut bus, host.as_mut())?; // one Thumb-2 instruction\n cpu.maybe_tick(&mut bus); // SysTick: time is instructions\n cpu.maybe_interrupt(&mut bus); // NVIC and PendSV delivery\n if cpu.cycles % 4096 == 0 {\n bus.pump_eth(host.as_mut()); // MGS packets in and out\n bus.pump_uart(host.as_mut());\n }\n}\n```\nNote: yes I’m aware I said there’s no cycles then have a field named cycles. That’s the \"fake it till you make it\" part of emulation.\n\nThat’s the overall heartbeat of sp-emu, and it runs ad infinitum. Everything else exists to make those calls seem real.\n\n## And One More Thing\n\nPast the actual working of the emulated sleds and microcontrollers, it’s all good ol'\nfashion user experience. When you’re running that many components on a single system, it\ncan get really confusing trying to figure out what pieces are where, how to connect\nto one sled on one rack versus another, etc. So the `voxel` command that we use to\nlaunch and destroy virtual racks also serves as a connectivity tool. For example:\n\n```\n$ pfexec ./target/debug/voxel host login g0\n[voxel] ssh root@192.168.68.167 (g0 global zone)\nThe illumos Project helios-3.0.23976 May 2026\nroot@g0:~# zoneadm list\nglobal\noxz_switch\noxz_internal_dns_5b07f3e7-444c-462c-9633-6967e52d6a3b\noxz_ntp_b308987c-3408-4077-b9a0-d803ed1e420a\noxz_cockroachdb_daa29fcd-fff0-4b3d-a722-b969e03398d9\noxz_cockroachdb_117986ee-8b2d-4567-8c6f-e01f3e63447e\noxz_crucible_310a91b5-1b67-46cb-b08d-ad6534702a3d\noxz_crucible_627c130b-97d6-4e8f-b13e-eb03b671c2a3\noxz_crucible_pantry_65e1b527-26fc-46fd-8584-d9792771926c\noxz_crucible_2692a74a-3a22-420c-857a-a3c786e4b600\noxz_external_dns_baa049eb-7bdd-4d8e-ace3-c0ef054845bd\noxz_crucible_2ffc1456-b0e1-4f8e-8b6d-6ab01377bd25\noxz_crucible_c5f7d004-9d9b-4741-9246-c4793b9c7976\noxz_nexus_c1a31871-bdb5-422c-b5b1-2de62aa31e30\n```\nThat command logs the user straight into the g0 sled in the global zone. Alternatively:\n\n```\n$ pfexec ./target/debug/voxel tp login switch0\n[voxel] ssh root@192.168.68.167 -> zlogin oxz_switch (g0 switch0)\n[Connected to zone 'oxz_switch' pts/3]\nLast login: Sun Sep 6 22:17:52 from fdb0:a840:2500:\nThe illumos Project helios-3.0.23976 May 2026\nroot@oxz_switch:~#\n```\n`tp` in this case acts like our [technician ports](https://docs.oxide.computer/guides/system/rack-installation#connect-tech-ports),\nand allows the user to connect straight to a switch zone for rack-wide commands and networking.\nThere are also numerous commands to manage things like images (export/import, delete), config\n(network topology, sled sizing, multi-rack, etc.), and more features coming as we build.\n\n[Voxel is available under the MPL 2.0 license](https://github.com/oxidecomputer/voxel/) for\ndownload. You’ll need to run Helios as your OS to run Voxel. I’d recommend 16+ threads\nand 64GB+ RAM, plus a good premium NVMe with onboard DRAM.\n","body_html":"

The trouble with testing and developing for rack-scale hardware is that rack-scale is not all that easy to come by. I mean, nevermind the whole DDR5-as-a-currency thing; even if all the parts were available in abundance, any completed racks go swiftly to the front lines (AKA our customers' data centers).

\n

As a member of the Product Assurance Team at Oxide, I find that totally unfair yet completely reasonable. And so we have to get a bit creative. We have to be able to:

\n\n

And the cherry on top: we need to do that without cutting special "test version" releases. Because we want to keep our CI sleek, but also because that wouldn’t be playing fair.

\n

Now, don’t get me wrong — we’re not without our tricks. When I joined Oxide last year,\none of the first things I was introduced to was our racklettes; that is, smaller\nthan average racks of four sleds and two switches each. These racklettes can be\ninitialized and run just like a real rack, precisely because they are a real rack.\nJust not in any form factor it would be economical to sell. But for testing they\nwork like a charm. We have four of them today, soon to be eight, where we can test\nany combination of hardware/software requirement.

\n

We also have our various simulated methods of running Omicron. You can read more about\nthose at How to Run Simulated\nand the full(er) How to Run Guide. I\nwon’t go into a ton of detail here, but suffice it to say that most parts of our control plane (and more) can be run independently, in tandem, and in various degrees of simulation.

\n

What’s a Voxel?

\n

Do you remember the game Q*bert? It was one of the first wildly popular isometric games, and looked a little like this:

\n

Why am I showing you a ridiculously small game screenshot? Well, each cube in that image represents a 'voxel,' or a point in 3D space. It’s actually the 3D version of a pixel. Volume Element + Pixel = Voxel. Neat, right?

\n

Anyway, for Oxide, Virtual OXidE Labs (or Voxel) means an incredible amount of fidelity packed into a single piece of hardware. It is the spiritual successor to our prior virtual labs (known as a4x2) which allowed us to hack out decent racks running in VMs.

\n

Pay attention to that word "fidelity" by the way.\nFor it is, as Bryan Cantrill puts it, the Prime Directive: labs should use\nstock Oxide software without hack or modification.

\n

This is what Voxel gives us. A complete software emulation stack, built on illumos in Rust, where we can build and test a variety of realistic configurations:

\n\n

And what’s more, it gives us all these features in a fairly small footprint. At home,\nI run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and\n96GB RAM. On that computer, I can spin up 6 sleds with ease, either in the form of\none rack or in a multirack configuration (two separate racks, one management plane). All\nit requires is Oxide Helios and a decently\nfast NVMe drive. Launch, and in about 10 minutes you have a full Oxide rack with each "sled"\nin its own Propolis VM — alongside switches (also in Propolis) and other infrastructure.

\n

Image Creation

\n

Remember how I said we have to use stock Oxide software? That actually makes things easier. The first thing we do (following a common practice in hypervisor tooling) is to create a reusable snapshot.

\n

Oxide sleds are only ever one of two things: a normal\nsled (a "gimlet" for the first generation, "cosmo" for the current) or a "scrimlet,"\nwhich is a sled wired to the sidecar switches that runs a switch zone. It’s all\namazingly elegant in its complexity, you should read more about it in\nour networking docs.

\n

To create a snapshot for Voxel, all we must do is create a "sled" (virtual machine), install our control plane (Omicron) on it, snapshot it before the point of configuration, et voilà: a ready-to-launch sled. The control plane can be installed one of two ways:

\n\n

Using the second option as an example, here’s the beginning of the process where voxel image create pulls out the key pieces that make up the sled config:

\n
$ pfexec ./target/debug/voxel image create --from-tuf ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip 5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c\n[voxel] TUF source ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip: system version 23.0.0-0.ci+git5d176e0be7e, omicron 5d176e0be7e\n[voxel] extracted 12 zones from ../5d176e0be7e5cc0d3288c1bef10005ec31f2cd7c.zip\n[voxel] extracted 3 measurement corpus artifacts\n[voxel] extracting host artifacts -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-host\n[voxel] host boot image 1258295296 bytes, phase 1 rom 33554432 bytes\n[voxel] extracted SP/RoT firmware -> /export/home/helios/voxel-rawfix/target/debug/../../voxel-image/.tuf-fw/ca9184058bd7
\n

After a decent amount of unpacking and staging, it will launch a VM that everything gets written/installed to. And finally:

\n
[voxel] ready: voxel-cp version=unknown unpacked_artifacts=18 built=2026-09-06T21:37:58\n[voxel] clearing device-instance map, halting\n[voxel] flushing, up to 25s\n[voxel] stopping hypervisor\n[voxel] writing voxel image voxel-cp-5d176e0be7e-tuf
\n

That image is a zfs snapshot which can be used and reused to create our rack.

\n

Ready for Launch

\n

Launching the rack uses a similar process, except the control plane is already installed. All we need at this point is:

\n
  1. An "frr" image (the Debian VMs for networking)
  2. A control plane image (like the one we just created)
  3. The voxel-init agent that gives each virtual sled its assignment
\n

The log output is pretty long, but you can scroll through it if you’d like. I’ll recap after.

\n
$ pfexec ./target/debug/voxel launch --emu\n[voxel] sled-agent config schema (image properties): data_links=Tagged disks=Hardcoded\nWriting key pair to "./cargo-bay/sprockets/test-root-a.key.pem"\nWriting key pair to "./cargo-bay/sprockets/test-signer-a1.key.pem"\n...\nWriting certificate to "./cargo-bay/sprockets/test-root-a.cert.pem"\nWriting certificate to "./cargo-bay/sprockets/test-signer-a1.cert.pem"\n...\nWriting pki path to "./cargo-bay/sprockets/test-sprockets-auth-0.certlist.pem"\nWriting pki path to "./cargo-bay/sprockets/test-alias-0.certlist.pem"\n...\nadd net fdb0:a840:2500:1::/64: gateway fdb0:a840:25ff:0::2\nadd net fdb0:a840:2500:5::/64: gateway fdb0:a840:25ff:0::4\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300\n[identity] seed = 0x2aac7ae3f8cfbb1e (3074967765137734430)\n[identity] SP UID 13120858064df80da1642f2b, RoT UUID 00a94a8e206f3f138a790fba1b12db36\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/sidecar.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/archives/sp.zip\nflashed 786688 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33300/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310\n[identity] seed = 0xa65abed0e20d08d1 (11987103162554976465)\n[identity] SP UID 72a3c3327682e0df38de478a, RoT UUID fcb600f7322f3393899f72036aa2b4fd\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33310/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320\n[identity] seed = 0xbab29f9c81fda354 (13452990531453690708)\n[identity] SP UID 0883f59f877ab40da88e8072, RoT UUID 15468fb5bffd30d3b381e8759621fb4f\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33320/sp-flash.bin\n[sp-emu] instance state dir: /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330\n[identity] seed = 0xea954465eef85936 (16903491980996663606)\n[identity] SP UID 49c0e331f4877c7d0afc71eb, RoT UUID 03ec945840463823a95d38dda9e261d1\n[flash] Hubris archive /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/gimlet.archive -> /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/archives/sp.zip\nflashed 728320 bytes into slot A (base 0x08000000, reset PC 0x080002e8) of /export/home/helios/voxel-unzip/sp-fleet/r0/sp-emu/state/33330/sp-flash.bin\n[voxel] rack 0 SP fleet up on fdb0:a840:25ff:0::1 (4 SP(s): [33300, 33310, 33320, 33330])\nSep 06 22:15:47.126 INFO launching runner: deployment(voxel) persistent(true) custom_propolis_binary(Some("/export/home/helios/propolis-server")) dataset(testbed/falcon) falcon_dir(.falcon)\nSep 06 22:15:47.126 INFO starting preflight for deployment voxel\nSep 06 22:15:48.162 INFO creating links\nSep 06 22:15:48.162 INFO creating simnet link 'voxel_ce_vn_sim0'\nSep 06 22:15:48.164 INFO creating vnic link 'voxel_ce_vn_vnic0'\n...\nSep 06 22:15:48.249 INFO creating external links\nSep 06 22:15:48.249 INFO creating external link voxel_ce_vn_vnic2\nSep 06 22:15:48.251 INFO creating external link voxel_g0_vn_vnic3\n...\nSep 06 22:15:48.255 INFO creating nodes\nSep 06 22:15:48.255 INFO g0: launching node\nSep 06 22:15:48.257 INFO g1: launching node\n...\nSep 06 22:15:48.290 INFO launched instance g0 with pid 13041 on port 43736\nSep 06 22:15:48.291 INFO g0: instance ensure\nSep 06 22:15:48.295 INFO launched instance g1 with pid 13042 on port 33187\nSep 06 22:15:48.295 INFO g1: instance ensure\nSep 06 22:15:48.297 INFO launched instance g2 with pid 13043 on port 51110\n...\nSep 06 22:15:52.423 INFO g1: instance ensure completed after 0 retries\nSep 06 22:15:52.423 INFO g1: instance run\nSep 06 22:15:52.479 INFO g2: instance ensure completed after 0 retries\nSep 06 22:15:52.479 INFO g2: instance run\nSep 06 22:15:52.534 INFO g0: instance ensure completed after 0 retries\nSep 06 22:15:52.534 INFO g0: instance run\nSep 06 22:16:03.961 INFO cr1: mounting /opt/cargo-bay\nSep 06 22:16:03.969 INFO cr2: mounting /opt/cargo-bay\nSep 06 22:16:03.994 INFO cr1: finished mounting /opt/cargo-bay\nSep 06 22:16:03.995 INFO ce: mounting /opt/cargo-bay\nSep 06 22:16:04.002 INFO cr2: finished mounting /opt/cargo-bay\n...\nSep 06 22:16:09.236 INFO ce: launch start\nSep 06 22:16:09.237 INFO cr1: launch start\nSep 06 22:16:09.237 INFO cr2: launch start\nSep 06 22:16:12.893 INFO ce: [voxel-init] pinned enp0s10 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.893 INFO ce: [voxel-init] NAT rack egress via enp0s10\nSep 06 22:16:12.894 INFO ce: [voxel-init] static edge IP 192.168.68.170/22 added on enp0s10\nSep 06 22:16:12.894 INFO ce: [voxel-init] router bring-up complete\nSep 06 22:16:12.894 INFO ce: launch ok\nSep 06 22:16:12.956 INFO cr1: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.956 INFO cr1: [voxel-init] NAT rack egress via enp0s11\nSep 06 22:16:12.956 INFO cr1: [voxel-init] router bring-up complete\nSep 06 22:16:12.956 INFO cr1: launch ok\nSep 06 22:16:12.985 INFO cr2: [voxel-init] pinned enp0s11 DHCP client-id to MAC; re-DHCPing host-LAN uplink\nSep 06 22:16:12.985 INFO cr2: [voxel-init] NAT rack egress via enp0s11\nSep 06 22:16:12.985 INFO cr2: [voxel-init] router bring-up complete\nSep 06 22:16:12.985 INFO cr2: launch ok\nSep 06 22:16:12.985 INFO g0: launch start\nSep 06 22:16:12.986 INFO g1: launch start\nSep 06 22:16:12.987 INFO g2: launch start\nSep 06 22:16:28.581 INFO g1: [voxel-init] underlay(jumbo)=["vioif0", "vioif1"] ext-candidates=["vioif2"]\nSep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0 (2FAKE001-M20): M.2 layout written\nSep 06 22:16:28.581 INFO g1: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:28.581 INFO g1: [voxel-init] c3t1d0 (2FAKE001-M21): M.2 layout written\nSep 06 22:16:28.582 INFO g1: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:28.582 INFO g1: [voxel-init] c4t1d0 (2FAKE001-U20): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c5t1d0 (2FAKE001-U21): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c6t1d0 (2FAKE001-U22): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c7t1d0 (2FAKE001-U23): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] c8t1d0 (2FAKE001-U24): labeled\nSep 06 22:16:28.582 INFO g1: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_626cc492-5a93-4b5c-833b-c45a504d8300)\nSep 06 22:16:28.582 INFO g1: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_711d472d-edbc-4426-9e33-4b000914f4da)\nSep 06 22:16:28.582 INFO g1: [voxel-init] sled-agent activated\nSep 06 22:16:28.582 INFO g1: [voxel-init] gimlet bring-up complete\nSep 06 22:16:28.582 INFO g1: launch ok\nSep 06 22:16:33.159 INFO g0: [voxel-init] underlay(jumbo)=["vioif1", "vioif2"] ext-candidates=["vioif0", "vioif3"]\nSep 06 22:16:33.159 INFO g0: [voxel-init] SP network fdb0:a840:25ff:0::2/64 on vioif3 (deprecated as a source)\nSep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0 (2FAKE000-M20): M.2 layout written\nSep 06 22:16:33.159 INFO g0: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0 (2FAKE000-M21): M.2 layout written\nSep 06 22:16:33.159 INFO g0: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:33.159 INFO g0: [voxel-init] c4t1d0 (2FAKE000-U20): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c5t1d0 (2FAKE000-U21): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c6t1d0 (2FAKE000-U22): labeled\nSep 06 22:16:33.159 INFO g0: [voxel-init] c7t1d0 (2FAKE000-U23): labeled\nSep 06 22:16:33.160 INFO g0: [voxel-init] c8t1d0 (2FAKE000-U24): labeled\nSep 06 22:16:33.160 INFO g0: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_ab4a1425-a7f2-47c7-9ef8-c70180ef02c9)\nSep 06 22:16:33.160 INFO g0: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_410f70f9-1eea-49fe-8aae-8331cebc4ea1)\nSep 06 22:16:33.160 INFO g0: [voxel-init] switch0 enforcer started (pid 769), log /tmp/switch-enforcer.log\nSep 06 22:16:33.160 INFO g0: [voxel-init] sled-agent activated\nSep 06 22:16:33.160 INFO g0: [voxel-init] gimlet bring-up complete\nSep 06 22:16:33.160 INFO g0: launch ok\nSep 06 22:16:38.179 INFO g2: [voxel-init] underlay(jumbo)=["vioif1", "vioif2"] ext-candidates=["vioif0", "vioif3"]\nSep 06 22:16:38.179 INFO g2: [voxel-init] SP network fdb0:a840:25ff:0::4/64 on vioif3 (deprecated as a source)\nSep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0 (2FAKE002-M20): M.2 layout written\nSep 06 22:16:38.179 INFO g2: [voxel-init] c2t1d0: host phase 2 seeded\nSep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0 (2FAKE002-M21): M.2 layout written\nSep 06 22:16:38.179 INFO g2: [voxel-init] c3t1d0: host phase 2 seeded\nSep 06 22:16:38.179 INFO g2: [voxel-init] c4t1d0 (2FAKE002-U20): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c5t1d0 (2FAKE002-U21): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c6t1d0 (2FAKE002-U22): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c7t1d0 (2FAKE002-U23): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] c8t1d0 (2FAKE002-U24): labeled\nSep 06 22:16:38.179 INFO g2: [voxel-init] 7 sled disks (2 M.2, 5 U.2)\nSep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c2t1d0s5 (oxi_270f44df-c4df-43fd-b02f-e5440857b547)\nSep 06 22:16:38.179 INFO g2: [voxel-init] preseed: staged install dataset on /dev/dsk/c3t1d0s5 (oxi_1c986103-d332-4270-b7e1-9341310a621d)\nSep 06 22:16:38.179 INFO g2: [voxel-init] switch1 enforcer started (pid 779), log /tmp/switch-enforcer.log\nSep 06 22:16:38.179 INFO g2: [voxel-init] sled-agent activated\nSep 06 22:16:38.179 INFO g2: [voxel-init] gimlet bring-up complete\nSep 06 22:16:38.179 INFO g2: launch ok\nSep 06 22:16:38.652 INFO rack-init: waiting for the commission API: switch-zone bootstrap address: no bootstrap address on oxz_switch\nSep 06 22:17:54.860 INFO rack-init: waiting for the commission API: tunnel opened\nSep 06 22:17:59.876 INFO rack-init: commission API reachable on g0\nSep 06 22:18:23.892 INFO rack-init: 3 bootstrap sleds ready\n..+.......+.....+.+..+.......+..+...............+....+.....+.+.....+......+....+...+..+....+.....+++++++++++++++++++++++++++++++++++++++*.....+.+.....+......+.+++++++++++\n............+..+.+..+...+.........+......+...+..........+++++++++++++++++++++++++++++++++++++++*...+...+..+++++++++++++++++++++++++++++++++++++++*..................+.....\n-----\nSep 06 22:18:24.023 INFO rack-init: commission-driven RSS started\nSep 06 22:18:24.027 INFO rack-init: watching RSS progress on the RSS node ...\nSep 06 22:18:24.228 INFO rack-init: polling RSS status via ssh root@192.168.68.167\nSep 06 22:18:32.410 INFO rack-init [9/16]: initializing internal DNS\nSep 06 22:18:49.186 INFO rack-init [11/16]: initializing NTP\nSep 06 22:19:13.745 INFO rack-init [12/16]: waiting for time sync\nSep 06 22:20:51.850 INFO rack-init [12/16]: waiting for time sync ... still working (1m in this step)\nSep 06 22:22:22.134 INFO rack-init [13/16]: waiting for database\nSep 06 22:22:47.334 INFO rack-init [14/16]: initializing cluster\nSep 06 22:22:55.805 INFO rack-init [15/16]: initializing zones\nSep 06 22:24:33.717 INFO rack-init [16/16]: handing off to Nexus\nSep 06 22:26:11.824 INFO rack-init [16/16]: handing off to Nexus ... still working (1m in this step)\nSep 06 22:27:49.894 INFO rack-init [16/16]: handing off to Nexus ... still working (3m in this step)\nSep 06 22:28:06.231 INFO rack-init: complete, rack initialized (RSS run b20a2f8d-e511-4a67-a666-7b74b2ce68d7)\nSep 06 22:28:06.231 INFO launch complete\nSep 06 22:28:06.250 INFO external route set: 198.51.100.0/24 -> 192.168.68.170 (ce)\nSep 06 22:28:06.292 INFO rack: external network reachable (dns 198.51.100.20)
\n

Like I said, a lot of steps. Here’s what happened:

\n
  1. Trust material is generated in the form of sprockets PKI
  2. Emulated STM32H753 and LPC55 microcontrollers are flashed with Hubris firmware
  3. Our automated network topology suite (known as falcon) wires the topology with simnet and VNIC pairs for every link, and stages the sleds (g0, g1, and g2 in this case)
  4. Propolis VMs launch; 6 total in this example, with 3 driving the network and 3 as Oxide sleds
  5. voxel-init runs: a lightweight Rust agent on each node that
  6. Connects to the commission API for wicketd , which brings the rack through its initialization phases
\n

It’s a complex dance of steps, and I really buried the lede on the whole "emulated\nmicrocontrollers" front. But suffice it to say these are the complete and right steps for\nany Oxide rack, except the hardware is represented as distinct pieces of code. For example,\nneed a new interconnect between two virtual racks? Add a couple lines of code in rack.rs.

\n

There are a few things that don’t follow the Prime Directive (yet!). For example, those certs\nthat make up the sprockets PKI up top aren’t set at manufacturing and coming from the RoT\nlike our production racks. But that’s one of those things that we can live with for now, as\nit doesn’t affect actual rack runtime.

\n

Emulating Service Processors

\n

This is the bit I mentioned a moment ago. A big part of what makes our racks unique is\nthe embedded RoT and SP\nin every sled. We’ve long had an SpSim construct that served as a stub for the service\nprocessor and would return the right info at the right times. But that’s simulation, not\nemulation. It’s always going to be happy path, and that’s just not enough fidelity for\nreal testing cases.

\n

As such, we built actual emulated Cortex-M7 and Cortex-M33 devices for illumos and Linux\n(x86 and ARM). The repo for this tool (sp-emu)\nis open source. Each of the two devices runs in a single thread, and (mostly)\nfaithfully emulates the three main requirements: instructions, time, and interrupts.

\n

From there, you can flash our Hubris firmware directly to these emulated devices, and then\ncommunicate by way of an ordinary socket. The Voxel lab can speak with these virtual devices via the\nManagement Gateway Service (MGS) and Inter-Processor Communication Channel (IPCC). You can learn\neverything you want and more about these concepts in the Management Gateway Service repo and Hubris docs.

\n

There really is no "run real microcontrollers as virtual devices with this one neat trick!"\nI can think of. But the real challenge of sp-emu’s job is time. Remember all of the code\nwe’re running there is meant for silicon. Hubris schedules off a 1ms SysTick and the emulator doesn’t\nuse wall time; as such, we had to invent a clock. That said, emulated hardware is also\nslower than the real thing. So there’s… well, less time per time. So instead of trying\nto get all temporal about it, time became instructions. For example, I give you maybe_tick:

\n
if self.systick == 0 {\n    self.systick = self.syst_rvr;\n}\nself.systick -= 1;\nif self.systick == 0 {\n    self.systick = self.syst_rvr;\n    self.tick_events = self.tick_events.wrapping_add(1);\n    if self.syst_csr & 2 != 0\n        && self.mode == Mode::Thread\n        && !self.primask\n    {\n        self.exception_entry(15, bus); // SysTick exception\n    }\n}
\n

In this case, we count instructions against the kernel’s own reload value and fire exceptions anywhere the single-priority design makes it safe to take it — in thread mode, interrupts unmasked. By the same token, peripheral access costs one tick of our clock (an instruction), whereas on silicon it might stall for dozens of cycles on the bus.

\n

You may have noticed self.exception_entry which is defined\nelsewhere in the code:

\n

fn exception_entry(&mut self, vecnum: u32, bus: &mut Bus)

\n

In short, this function models everything the hardware does on a Cortex-M core taking\nan exception, all the way between "exception becomes active" to the first handler\ninstruction. vecnum here is an ARMv7-M exception number (also the vector table index\nfor that exception’s handler). bus is the memory system which is used to actually\npush the frame and read the table.

\n

Hubris has a non-preemptible kernel, though it schedules tasks preemptively. SVCall,\nPendSV, SysTick and all IRQs run at the same\npriority level, while faults keep a higher priority. As such, the emulator doesn’t have\nto nest handlers. It processes pending IRQs based on the PRIMASK, FAULTMASK, and BASEPRI\nsettings. If one of them pended PendSV, that will perform the deferred context\nswitch for the kernel. For more details on how it works internally, you can check\nthis lovely bit of documentation.

\n
if let Some(irq) = bus.next_irq() {\n    if self.basepri == 0 || (bus.irq_prio(irq) as u32) < self.basepri {\n        bus.clear_pending(irq);\n        self.exception_entry(16 + irq as u32, bus);\n        return;\n    }\n}\nif bus.take_pendsv() {\n    self.exception_entry(14, bus);\n}
\n

There’s a lot more we can talk about here, but to sum it all up you can look at this pseudo-ish loop:

\n
loop {\n    cpu.step(&mut bus, host.as_mut())?;   // one Thumb-2 instruction\n    cpu.maybe_tick(&mut bus);             // SysTick: time is instructions\n    cpu.maybe_interrupt(&mut bus);        // NVIC and PendSV delivery\n    if cpu.cycles % 4096 == 0 {\n        bus.pump_eth(host.as_mut());      // MGS packets in and out\n        bus.pump_uart(host.as_mut());\n    }\n}
\n

Note: yes I’m aware I said there’s no cycles then have a field named cycles. That’s the "fake it till you make it" part of emulation.

\n

That’s the overall heartbeat of sp-emu, and it runs ad infinitum. Everything else exists to make those calls seem real.

\n

And One More Thing

\n

Past the actual working of the emulated sleds and microcontrollers, it’s all good ol'\nfashion user experience. When you’re running that many components on a single system, it\ncan get really confusing trying to figure out what pieces are where, how to connect\nto one sled on one rack versus another, etc. So the voxel command that we use to\nlaunch and destroy virtual racks also serves as a connectivity tool. For example:

\n
$ pfexec ./target/debug/voxel host login g0\n[voxel] ssh root@192.168.68.167  (g0 global zone)\nThe illumos Project     helios-3.0.23976        May 2026\nroot@g0:~# zoneadm list\nglobal\noxz_switch\noxz_internal_dns_5b07f3e7-444c-462c-9633-6967e52d6a3b\noxz_ntp_b308987c-3408-4077-b9a0-d803ed1e420a\noxz_cockroachdb_daa29fcd-fff0-4b3d-a722-b969e03398d9\noxz_cockroachdb_117986ee-8b2d-4567-8c6f-e01f3e63447e\noxz_crucible_310a91b5-1b67-46cb-b08d-ad6534702a3d\noxz_crucible_627c130b-97d6-4e8f-b13e-eb03b671c2a3\noxz_crucible_pantry_65e1b527-26fc-46fd-8584-d9792771926c\noxz_crucible_2692a74a-3a22-420c-857a-a3c786e4b600\noxz_external_dns_baa049eb-7bdd-4d8e-ace3-c0ef054845bd\noxz_crucible_2ffc1456-b0e1-4f8e-8b6d-6ab01377bd25\noxz_crucible_c5f7d004-9d9b-4741-9246-c4793b9c7976\noxz_nexus_c1a31871-bdb5-422c-b5b1-2de62aa31e30
\n

That command logs the user straight into the g0 sled in the global zone. Alternatively:

\n
$ pfexec ./target/debug/voxel tp login switch0\n[voxel] ssh root@192.168.68.167 -> zlogin oxz_switch  (g0 switch0)\n[Connected to zone 'oxz_switch' pts/3]\nLast login: Sun Sep  6 22:17:52 from fdb0:a840:2500:\nThe illumos Project     helios-3.0.23976        May 2026\nroot@oxz_switch:~#
\n

tp in this case acts like our technician ports,\nand allows the user to connect straight to a switch zone for rack-wide commands and networking.\nThere are also numerous commands to manage things like images (export/import, delete), config\n(network topology, sled sizing, multi-rack, etc.), and more features coming as we build.

\n

Voxel is available under the MPL 2.0 license for\ndownload. You’ll need to run Helios as your OS to run Voxel. I’d recommend 16+ threads\nand 64GB+ RAM, plus a good premium NVMe with onboard DRAM.

","headings":[{"level":2,"text":"What’s a Voxel?","id":"what-s-a-voxel"},{"level":3,"text":"Image Creation","id":"image-creation"},{"level":3,"text":"Ready for Launch","id":"ready-for-launch"},{"level":2,"text":"Emulating Service Processors","id":"emulating-service-processors"},{"level":2,"text":"And One More Thing","id":"and-one-more-thing"}]}}