Sep 21, 2026 Jay Lux Ferro agentsllmllm-agentspythoninfrastructurebenchmarks llm-agents For months, a 1-billion-parameter model validated sensitive-data detections in our proxy pipeline. Last week we ran it against a 40-line Python function. The model kept 85% of the deliberately invalid test data — fake credit cards with broken checksums — and quietly dropped real IBANs, real social security numbers, and a person’s actual email address. The function caught every fake, kept every real one, and answered in ten microseconds.
We didn’t set out to pick a fight with local LLMs. We set out to answer one question — can each layer of our pipeline do the same job, or better, without a local model? — and we refused to flip any switch until a benchmark said yes. What follow are the four experiments, the numbers, and the pattern hiding underneath them. The pattern matters more than the numbers.
The setup
Manifold chains proxy services — a redactor, a rate limiter, a compaction layer, a memory layer — between coding agents and cloud APIs. Over months of accretion, a local Ollama had snuck into nearly every layer, each instance reasonable at the time:
- the redactor asked a small model to double-check PII detections before scrubbing them,
- the memory layer embedded your request and ranked graph nodes by cosine similarity,
- the compaction layer asked a model to summarize old conversation turns,
- the cache layer embedded queries to detect repeats.
Four different jobs, one shared assumption: a model is the right tool for judging text. We decided to test the assumption job by job, with a hard rule: no mode ships without a re-runnable benchmark, and the benchmark has to show same-or-better before we change anything. Every table below is from those benchmarks, on identical inputs.
Experiment 1: Checksums vs. judgment
PII validation is the clearest case. When the redactor detects something that looks like a credit card, should you scrub it? The old answer: ask the model. The new answer: run Luhn.
The benchmark corpus was 120 labeled spans — valid cards, digit-flipped invalid ones, real-format IBANs with broken mod-97 checksums, SSNs with impossible area codes, malformed emails, truncated JWTs. Three arms over the same corpus: no validation, the model (llama3.2:1b, temperature 0), and a rules backend (Luhn, ISO 13616 mod-97, SSA area/group/serial rules, shape checks):
| arm | invalid dropped | valid kept | latency (p50) |
|---|---|---|---|
| none | 0% | 100% | 0 ms |
| model | 14.3% | 89.7% | 80 ms |
| rules | 100% | 100% | 0.01 ms |
Read the middle row carefully, because it’s worse than it looks. The model wasn’t merely weak — it was wrong in both directions at once. It waved through nearly every invalid span, and it also dropped eight genuinely valid ones: two IBANs, two SSNs, an email address, a person’s name. Those are leaks and false alarms in the same run, from the same component. And a detail we only found while building the replacement: the model had never even seen regex-detected spans — the original design only routed NER detections to it. The rules validate everything.
There’s an asymmetry that makes this durable. When Luhn rejects a card number, you can point at the exact digit that failed. When the model rejects an IBAN, you get a shrug. One of these failure modes is debuggable; the other is only averageable.
Experiment 2: Selection vs. paraphrase
The compaction layer summarizes old conversation turns so agents stop resending their entire history. Summarization is supposed to be the LLM’s home turf. We built the alternative anyway: a frequency-scored sentence selector that picks the most load-bearing sentences and joins them verbatim.
Same corpus, both backends, identical gates:
| backend | chars saved | fidelity | latency |
|---|---|---|---|
| model (llama3.2) | 40.9% | 0.909 | 1312 ms |
| extractive (CPU) | 67.2% | 1.000 | ~0 ms |
The fidelity column is the interesting one, and it’s a category error in the model’s favor that it still loses. The extractive summary’s fidelity isn’t a score — it’s a theorem. Selection cannot invent or paraphrase anything; the output is a subset of the input, so file paths, commit hashes, and error messages survive by construction. The model drifted to 0.909 doing the one thing it was asked to do gently, and it compressed less while doing it. When your invariant is “never lose the identifiers,” a system that structurally cannot lose them beats a system that mostly doesn’t.
Experiment 3: Token overlap vs. embeddings
The memory layer ranked knowledge-graph nodes against your request by embedding both and multiplying in a cosine-similarity factor. The lexical replacement is almost embarrassingly simple: tokenize the request, tokenize each node’s canonical form, multiply by the overlap fraction. No model, no vector store round-trip, no 150 ms timeout guarding a cold model.
Four-arm ablation on the benchmark harness (Anchor Jaccard measures whether the injected context picks the same nodes as the ground truth):
| arm | applied | p50 latency | anchors J | related J |
|---|---|---|---|---|
| off | 0% | 0.0 ms | 0.33 | 0.33 |
| structural | 100% | 3.9 ms | 1.00 | 0.44 |
| embedding | 100% | 33.9 ms | 1.00 | 0.44 |
| lexical | 100% | 2.4 ms | 1.00 | 0.44 |
Identical selection. Fourteen times faster. The corpus explains it: the things being matched are canonical forms like fastapi and path:src/lattice/server.py. Identifiers don’t have synonyms. Embeddings are insurance against paraphrase — and paraphrase is precisely what token-shaped text doesn’t have. We were paying a model to protect us from a risk that couldn’t occur.
Experiment 4: The one we lost
Honesty check. Our cache layer detects repeated questions: embed the query, compare against stored vectors at 0.95 cosine. The lexical replacement is Postgres trigram similarity — no embeddings, no model. On exact repeats it ties the embedding backend (both 100% hit rate) at 0.22 ms vs 18.9 ms per lookup. But on paraphrased repeats — the same question genuinely reworded — embeddings found 40% and trigrams found 20%.
We kept the embedding backend as the default for that layer and documented the trade in the preset that disables it. This is the experiment that keeps the other three honest: it’s not that deterministic methods always win. It’s that you don’t know which of your LLM calls are load-bearing until you try to replace them — and in our stack, three out of four weren’t.
The pattern: shape of signal, shape of tool
Line the experiments up and the pattern is about the shape of the signal, not about models being bad:
- Checksum-shaped (credit cards, IBANs, SSNs, JWTs): there is mathematics here. Use the mathematics.
- Subset-shaped (summarize without losing facts): selection gives you structural guarantees; generation gives you vibes with a similarity score bolted on.
- Token-shaped (identifiers, paths, API names, log lines): lexical matching is the native signal; embeddings sell you synonyms you can’t use.
- Paraphrase-shaped (humans rewording the same question): embeddings genuinely win. We measured it and kept them.
- Shapeless (rewrite this so it doesn’t sound like you, answer this open question): that’s what the model isfor .
Most agentic and infrastructural text is token-shaped or checksum-shaped. LLMs are the right tool when the input has no shape. Our mistake wasn’t using models — it was using them on problems that had already been solved, deterministically, decades ago, and paying model prices for engineering problems in costume.
Determinism is a feature you can measure in someone else’s bill
Two war stories from the same engineering arc, both about why determinism compounds.
The first: our memory layer injected a context brief at byte zero of every request. Its selection walk was bounded by a wall-clock deadline — and the same request, sent three times, produced three different outcomes (skip, inject, degrade) depending on how fast Postgres answered. Because the brief lives at the start of the prompt, every flip invalidated the provider’s prompt cache for the entire session. A latency race in one layer was quietly taxing every downstream token. We replaced the clock with a page budget — a pure function of graph state — and byte-stability returned. Caching layers punish nondeterminism at a distance; you’ll find out from your invoice, not your logs.
The second: one malformed SQL statement — a single trailing comma — took our entire telemetry pipeline down for an evening, and the test suite never noticed, because the tests recorded SQL strings without ever parsing them. A deterministic system fails somewhere specific, which means a bisect finds it. Our delegate agent found the comma in minutes with a prefix search. Stochastic failures don’t bisect; they regress to the mean and come back next Tuesday.
Where this leaves us
The mesh now runs its redaction, ranking, summarization, and caching layers with zero local-model dependency — each behind a config flag, each with the model path still there, each with a benchmark that must pass before anyone flips a flag back. The stack didn’t lose capability; it lost a dependency, a latency tax, and a class of failures that only show up in aggregate.
The question I’d leave you with isn’t “LLM or no LLM.” It’s the one we now ask before any model call: what is the shape of this signal? If you can write a checksum, a tokenizer, or a selection rule for it, you don’t have a learning problem. You have an engineering problem that learned to imitate one.
The most reliable component in our stack is now a function with no dependencies that has never surprised us. In infrastructure, that sentence is the whole review.
Part of the series Llm agents