First principles#

I don't hear people talking about first principles as often as I used to, but they're still important to me and to how I make decisions. When building the first version of Antfly, the idea was to fill a gap in the database market: a schema-ish, friendly query engine for indexing like Elasticsearch, closer in scale to Postgres than Iceberg, as easy to use as Mongo, and as easy to operate as Google's Spanner or Bigtable. (And I thought perfect search was too lofty... HA!)

When building the first version I didn't have the same sort of software tools (cough cough Codex, Claude, Aider, Pi) available to assist in development, and the hard problems I wanted to tackle were the ones CockroachDB had similarly chosen Go for: distributed systems and concurrency. Rust can guarantee memory safety through lifetimes, but memory safety wasn't the hard part, and Go is infinitely more readable to me than Rust ever was. Plus, Go had the most battle-tested Raft implementation out there (etcd's), and I was only one person, working on weekends and on my Fridays, trying to build an Elasticsearch DSL on top of Bleve as well.

Caring about the model, not the embeddings#

Fast forward a little bit and embeddings started to become a thing. word2vec had shown that a vector could actually carry meaning, and the first practical embedding models meant an average dev could build Google-lite semantic search for their app. At work, our scale of vector storage was so small that a top-k could be an exhaustive search, but I read about all the fun algorithms behind Pinecone, Vertex AI Vector Search, Elastic, Mongo, CockroachDB, and pgvector after an old coworker rolled his eyes at the ridiculous Pinecone seed round. Pinecone might have been overvalued, but I think one of its most interesting ideas was overlooked: users could care about the model instead of having to care about the embeddings. It made me think about Postgres, and how for the most part a user can avoid knowing about B-trees and other indexing algorithms, or how in Bleve and Lucene you can avoid knowing about S2 indexes, finite state transducers, and so on. Vector databases and indexes, on the other hand, required you to know and care about HNSW, SPFresh, RaBitQ, and the rest.

I decided to try my hand at implementing these algorithms from their papers and blog posts in the initial version of Antfly, and it was wildly successful. I was able to make a semantically searchable Wikipedia using my laptop, Antfly, and Ollama! Embedding generation was so slow that the database being a little bit slower was not a big concern for the initial implementation.

Enter Zig#

Concurrently with all this, I had tried my hand at implementing VSR, Protobuf, and an LSM in Zig a few years prior, after stumbling upon TigerBeetle. I really liked the readability of Zig, the concurrency primitives, the people implementing the language, and the interoperability with C. But Zig was too green for my weekend database project and lacked a lot of the heavy lifting: Raft, full-text indexing, a portable, battle-tested LSM, yada yada yada. I took a lot of the spirit of the TigerBeetle folks with me, though, and put myself to work incorporating VOPR testing with TLA+ trace validation (Rowan's post on formal verification with coding agents covers that), built on the new Go mock time and on prior art from etcd's Raft TLA+ spec and trace validation.

Then std.Io started making a big splash across the technoverse when Zig decided to make some serious overhauls to the language. I had been reading the GitHub design threads on the subject for a while and thought it was all pretty cool from an engineering perspective. Antfly had just released v0.1 and I had a breath of air to start thinking about what came next. So I started to play around with Zig and the new std.Io work, trying to implement our LSM using the async I/O and std.Io.Evented machinery Zig had started to expose. At the same time, I wanted to see how far I could take the software factorization of our code, and it felt like Raft, TLA+ specs, clear traces, and language-agnostic tests were the perfect hill to climb. So I went to work directing traffic and building Raft, full-text indexing, an LSM, and... HTTP/2 (and our Raft transport, which was over HTTP/3 and QUIC). I had also rewritten most of our Go-based end-to-end tests in Python, both to make sure the coding agents couldn't "cheat" by reaching into the Go code and to make sure our Python SDK was solid. This turned out to be the perfect language-agnostic framework for transpiling to Zig too.

Why rewrite#

A conversation with James and another with Drew really solidified the idea for me when we talked about what we were optimizing for: asymmetric outcomes and reliability. Most startups land on their face, so the expensive bet, writing every high-performance dependency ourselves, is the one worth making. A user MUST know and trust that their database works, and a user wants that database to fly, not sprint (it needs to be astonishingly fast). People were already building all sorts of interesting projects on top of Antfly. To run it from any other programming language or in the browser, though, we'd need something like Rust or Zig to give the code a C-compatible, WASM- and WebGPU-compatible interface. I wanted Antfly to be the grand unified theory of databases: a machete for old-school use cases and traditional apps, and the perfect Swiss army knife for the AI and semantic use cases nobody has seen yet (perfect search didn't seem lofty enough anymore). Something a developer could embed in a sandboxed environment (laptop, unit tests, Lambda), run at average application workloads (PostgreSQL, Cockroach, Mongo), or run at analytic scale (data warehouse, serverless). And I wanted Antfly to be even more reliable in all of those environments while maintaining or exceeding the performance goals we had set for ourselves. A rewrite of everything from the ground up gives you a chance to build all your dependencies in a purpose-aware shape, baking in resource management, priority scheduling, and testing ideology from day one.

So why Zig and not Rust? Four reasons, all back to first principles: portability, C interop, speed, and testability.

Portability and C interop: Zig is a C compiler with a libc for every target. Cross-compiling Antfly with CUDA, ONNX, and Wasmtime linked in is one flag, and calling them is @cImport. Rust cross-compiles pure Rust fine and breaks on the first C dependency, which is why cargo-zigbuild is Zig.

Speed: same LLVM, so codegen is a wash. The difference is the fast version is the default. Every allocation takes an allocator, nothing allocates or branches behind your back, and SIMD and comptime specialization are built in. In Rust the hot paths end up in unsafe. Next to Go it's not close: no GC, no scheduler, no CGO.

Testability: std.Io makes the world a parameter. Hand a package a simulated Io and it gets VOPR: disk, network, and clock faults, all of it. In Rust you rip out tokio for madsim or turmoil and hope the dependency tree cooperates.

People usually think formal verification and memory safety protect you from a whole class of bugs, and they do, but much like the validity of your lifetime in Rust, it depends on the context... Rust has a borrow checker and Zig doesn't, but the bugs that bug me aren't use-after-frees. They're a replica that fell behind, a message that arrived twice, an fsync that lied. The borrow checker doesn't cover that, and tools like Kani and Verus prove things about Rust functions, not about whether your Raft is linearizable. Rust or Zig, you still need something like Antithesis throwing faults at the whole system. In either language you want every bounds and overflow check on while you test, because one missed check is someone's data. Rust turns overflow checks off in release like everyone else; Zig makes safety one build mode you can flip. So we verify the protocol, TLA+ specs, trace validation against the running Raft, and VOPR on every package, none of which cares what language you wrote it in. Memory safety we buy in testing: every suite runs in Debug and ReleaseSafe with the testing allocator and the same simulator, and we ship ReleaseFast.

There was one more big reason to rewrite that I had wanted from the Go version anyway. We had already started to own forks of all our major dependencies. If we owned every dependency outright, we could control memory, CPU, and GPU resources holistically across the whole process, the same argument TigerBeetle makes in Tiger Style with its zero-dependencies policy. With Zig, zero dependencies is something you can actually keep, not a purity goalpost that moves every time you need something to go faster. Linking a C library is a non-event, so CUDA, Metal, and Wasmtime cost a few lines each instead of the toll CGO charges every time you cross the border. So we still use native code where it's the right tool, we just never let it run the show. Outside libraries snap into our engine, they don't get to wrap it, and everything that touches memory, threads, storage, and the network is ours. Writing an LSM, a B+tree, Raft, and HTTP/3 from scratch is a terrible trade for a couple of shiny new features, but a great trade for being able to fly instead of walk.

Some things made the rewrite harder: we had already made the Go version pretty freaking reliable, and bonkers fast for a Go program when comparing ourselves to our competitors. Zig is pre-1.0 and the standard library moves. The tooling is younger and fewer people can read the code. And I was also about to start a family while trying to get a startup off the ground, which is not an easy recipe for success.

What we got#

So that's why we did the rewrite, but what did we get out of it? First, the thing I wanted most. We were able to bake VOPR simulation testing in from day one, in all our dependencies and every package we built. Full control of the world: Antfly became the node, and std.Io was the perfect environment to plug into so that we could simulate system, network, concurrency, and clock faults everywhere and turn simulation testing into just... testing. We could tune the LSM for our vector-heavy, batch-oriented workloads and make the LSM's caching play nicely with the caching of the full-text and vector indexes.

Then there's everything else we were able to build in from the ground up, laying an amazing foundation for our vision.

Data engines#

• Document (Mongo, Elasticsearch): schema-ish JSON documents with columnar stored fields and projection pushdown
• Relational (Postgres): closed schemas and typed packed rows, with full-text, vector, sparse, graph, and algebraic indexes derived over the same base store
• Lake (Iceberg, Parquet): external tables queried in place over object storage, with the same derived indexes materialized as sidecars
• Foreign sources: stream tables out of PostgreSQL via logical replication

Storage engines# ModeLikeAutoscalingDurability / HASingle-file (Lite, .aflite)SQLiteverticalfsyncSingle-nodePostgresverticalhot standby with fencingDistributedCockroachDBvertical + horizontalmulti-Raft, online shard splits, cross-shard transactionsServerlessNeon, Lakebasevertical + horizontalobject storage

Under all four: our own LSM, an LMDB-compatible B+tree, a WAL, TTL, and portable backup and restore, each with its own simulation harness.

Indexing engines#

• Full text (Lucene, Elasticsearch): Lucene-style segments, Snowball stemmers, stopwords; geo, keyword, n-gram, numeric, wildcard, regex, and fuzzy queries; highlighting and aggregations
• Dense vector (Weaviate, pgvector): RaBitQ compression with SPFresh-style incremental posting maintenance, filtered by the other indexes
• Sparse vector: SPLADE learned sparse retrieval alongside BM25
• Late interaction: ColBERT-style MaxSim over multi-vector documents (ColQwen2)
• Hybrid fusion: reciprocal rank fusion, relative score fusion, and result pruning across all of the above in one query plan
• Algebraic: aggregates and materializations (sum, min, max)
• Graph (Neo4j): traversal, pathfinding, PageRank-style queries, ad hoc foreign keys and views

Enrichment engines#

One mechanism: an asset producer turns a raw field into a durable artifact, and any index can source from that artifact. Artifacts are reprocessable, readiness is tracked per stream, and nothing is silently dropped.

• Any input shape: text, JSON, PDFs, images, audio, and video, by URL or inserted directly
• Documents: born-digital PDF parsing, with Florence-2 OCR for pages that carry no text, and LayoutLMv3 for document layout
• Audio: Whisper transcription into searchable transcripts, and CLAP for native audio embeddings
• Images: CLIP and Gemma multimodal embeddings, so text queries retrieve images and image queries retrieve text
• Chunking: fixed and learned chunkers, with document-level and chunk-level vectors in one index
• Unstructured to structured: GLiNER2 entity extraction, classification, and summarization write typed fields you can filter, aggregate, and join on
• Autograph: a graph index sourced from an extraction artifact, so entity and relation edges materialize as documents arrive, with no separate graph load step
• Bring your own: application-managed vectors and precomputed descriptions plug into the same indexes

Query engines#

• HTTP JSON API: Lucene, Elasticsearch, Bleve, and Mongo-style DSL, with streaming RAG over SSE
• SQL and the Postgres wire protocol: DDL, CTEs, joins, window functions, RETURNING, and search_path sessions, lowered to native typed plans rather than SQL text, plus a document SQL dialect over JSON tables
• Agents: retrieval agent, query-builder agent, agent tools, and MCP and A2A surfaces

Embedding surfaces#

• Single binary, Kubernetes operator with autoscaling, or Antfly Cloud
• C API and antfly-embedded for in-process use from other languages
• WASM builds of the inference runtime (wasm32 and wasm64)
• WASM extension runtime (Wasmtime) for user code inside the engine

Compute backends#

• CPU (SIMD)
• Metal
• CUDA, with a kernel JIT

Model runtime#

• Formats: GGUF, safetensors, ONNX; native quantization, export, and compiled artifacts
• Architectures: BERT, ModernBERT, NomicBERT, DeBERTa, T5, GPT, Qwen2/2-VL/3-VL, Gemma 3/4 (multimodal), Whisper, Florence, LayoutLMv3, GLiNER
• Tasks: embedding (text, image, audio, late interaction), chunking, reranking, classification, NER, OCR and layout, transcription, generation with tool calling
• Remote providers: OpenAI, AWS Bedrock, Vertex, Ollama

Fine-tuning#

• Methods: LoRA, QLoRA (NF4), recursive LoRA, PEFT; SFT, DPO, ORPO, SimPO, GRPO, RFT; NEFTune, gradient checkpointing, sequence packing
• Recipes: Gemma 4, GLiNER2, ColQwen2, LayoutLMv3, reranker heads and LoRA, fused SPLADE chunker

Verification#

• VOPR simulation harnesses for the LMDB backend, LSM, WAL, transactions, Raft, metadata, and index manager
• TLA+ trace validation for Raft
• Language-agnostic Python end-to-end suite shared with the Go version

Beyond correctness#

In addition to rigorous correctness testing, we've been working hard on effectiveness, usability, and performance testing. Model cards are the new hotness for summarizing the big, powerful capabilities of LLMs, and we've borrowed the idea for our database releases: the Antfly 0.2 release card is our model card. Rowan has done a bunch of great work evaluating Antfly's performance across a variety of benchmarks and domains, against a variety of competitors that each specialize in one of those domains. We've been evaluating Antfly as a solution for RAG, OCR, and memory systems. We've also been building new benchmarks and evals to show you where Antfly really shines: the ability to implement the features you care about, effortlessly scaffold your database use cases and scale them, and discover the features and capabilities you need to make retrieval better now that AI produces more information and assets than ever.

Perfect search is still silly, ambitious, and unobtainable. We're just a lot closer than we were in Go, and this time we won't have to rewrite the foundation to get the rest of the way.

If you want to see all of this running, the Quickstart installs Antfly, loads 10,000 Wikipedia articles, and walks through full-text, semantic, hybrid, and image search and RAG, with every step in the CLI, cURL, TypeScript, Python, and Go.

Links#

• Quickstart: antfly.io/docs/guides/quickstart
• Release card: antfly.io/releases/v0.2
• GitHub: github.com/antflydb/antfly/tree/main/zig
• Docs: docs.antfly.io
• Site: antfly.io
• Discord: discord.gg/zrdjguy84P