TypeSafe AI's Jev\nhighlighted a useful idea: software needs decisions, not more strings to parse.\nJev answers it with a purpose-built model. TypeAR asks the complementary\nquestion — whether the same typed-decision interface can come from any open\nautoregressive model you already run.
\nSo that is what it is: you hand TypeAR context and an ordered JSON Schema, and it hands back values your software can act on — enums, booleans, scores — with no proprietary model API, no retraining, and no parser between the model and your code.
\nThe guarantee it makes is narrow and absolute: a returned value is always\none of the values your schema declared. Not usually. Not after a retry. There\nis no validator in the hot path, because there is nothing for a validator to\nreject.
\nStart with one field of the schema:
\nTypeAR renders that field into the prompt and stops mid-literal, exactly where the answer belongs:
\nThe model completes one token there, and two things make that enough.
\nMap every value to one token. meal, travel and equipment tokenize to\ndifferent lengths, so they cannot be compared at one position. Each gets a\nsingle-token control label — A, B, C — written on the choices line.\nLabels are checked against the server's tokenizer: one token in, the same text\nback out, or the label is skipped.
Only those labels are selected. TypeAR reads the log probability of each\ncandidate token and nothing else, so an out-of-schema token can never be chosen:\nit was never a candidate. Renormalizing over the candidates gives a\ndistribution on your declared values — meal 0.04, travel 0.93, equipment\n0.03 — which sampling draws from and you can threshold on. Argmax does not need\nit: restricting the candidates already fixes the ranking.
All of this needs the model's per-token probabilities, which closed APIs do not\nexpose. TypeAR gets them from an open-source model served with\nSGLang.
\nOne forward pass covers a decision however many values it allows, so a boolean\nand an eleven-level score cost the same. Temperature acts on the candidate\nscores, never on free generation: turning it up makes meal likelier than\ntravel, never something the schema never declared.
Everything above describes one decision. A schema is several of them, and there are two ways to run the set. They differ in exactly one thing: what each decision is conditioned on.
\nThe selected label goes back into the same string along with the value it stands\nfor, so the next decision reads value="travel" instead of having to remember\nwhat B meant. Then the next question is appended: field order is decision\norder. The second request is therefore the first request, character for\ncharacter, plus a tail:
Everything above value="travel" was already prefilled for the first decision\nand is byte-identical now, so the server's prefix cache matches it and reuses\nthat KV state. The client holds one growing string and never touches a KV\ntensor; only the new tail is computed.
The server reports how many tokens it reused. With D decisions, a context of\nC tokens, and roughly S new tokens per decision:
Not every schema needs that. Sixteen independent yes/no flags about one document are answerable from the context alone — nothing in flag 12 depends on flag 11 — and running them in sequence makes each wait for a round trip it never needed.
\nSo TypeAR forks instead of extending. The shared context is prefilled once and\neach field becomes a branch: the same prefix plus its own record, ending at the\nsame answer=" cut. All branches go out in one call and come back as a single\nbatched decode step, one token per field.
Because every branch starts with identical text, the prefix cache serves all of them. In one local run — a ~1,100-token shared context, sixteen boolean fields, one GPU, sixteen concurrent requests allowed — every branch reused 1,088 cached tokens:
\n| Execution | End-to-end | Per decision | Relative |
|---|---|---|---|
| Sequential | 9.35 s | 0.584 s | 1.0x |
| Batch | 1.61 s | 0.101 s | 5.8x |
One measurement on one machine, not a portable benchmark; it moves with K, the\nmodel, the context length and the server. The shape is the durable part: prefill\nof about C + sum(Q_k) for K fields, then one batched decode.
The cost is the dependency — each branch sees only the context and its own question — so use batch when the fields genuinely are independent. Everything else holds: each branch still stops mid-literal, still scores only single-token labels, still renormalizes over its own declared domain.
\nThe two modes are not a speed knob. They compute different conditionals, so the\nquestion is never "which is faster" but "does this field's answer depend on\nanother field's answer". A severity that should follow from the system already\nidentified, a rollback that should follow from both — those belong in sequence,\nand the cost is D round trips. Sixteen flags that each read the same document\nindependently belong in one batch, and the cost is that none of them can see the\nothers.
Getting it wrong in one direction is slow; in the other it is wrong. A field batched away from a dependency it actually has will answer as though that dependency did not exist, and nothing in the output will say so. Nothing stops you from splitting a workflow either — run the dependent fields in sequence, then batch the independent ones over a context that already carries the first results.
\nEvery decision so far has come from a finite set, because that is what single- token labels can cover. The schema layer enforces it: an enum, a boolean, or a score field that expands to eleven levels between 0.0 and 1.0. A typed field without an enum is rejected outright rather than guessed at.
\nNothing in the mechanism requires that, though, and this is where a decoding method has room that a purpose-built model does not. Declare one branch of the field open:
\nThe typed decision is unchanged — four single-token candidates now instead of\nthree. Pick A, B or C and the value is written back as before. Pick D\nand the record opens a string rather than closing one, and generation continues\nunconstrained until the closing quote:
The guarantee survives because the escape is itself a declared value: free text\nis reachable only through a token you put in the domain, and the closing quote\nbounds what comes out of it. Nor does what comes out have to stay a one-off —\nthe domain is just a list written into the prompt, so conference registration\ncan be bound to its own label and offered as a declared choice in the next\ndecision. The vocabulary grows from what the model produced, and every later\ndecision is still one token over a finite set.
TypeAR shows that a typed-decision interface does not require a purpose-built model. Restricting the candidates at one position, renormalizing over them, and appending the result to the prompt is enough to get values that are typed by construction — one output token per decision, one read of the context per workflow, from an open model you already serve. No proprietary API and no retraining: it reads a distribution the model already produces.
\nMore benchmarks and worked examples are coming.
","headings":[{"level":2,"text":"How a type-safe answer is generated","id":"how-a-type-safe-answer-is-generated"},{"level":2,"text":"Two execution modes","id":"two-execution-modes"},{"level":2,"text":"Sequential: each decision sees the last","id":"sequential-each-decision-sees-the-last"},{"level":2,"text":"Batch: independent fields in one request","id":"batch-independent-fields-in-one-request"},{"level":2,"text":"Sequential or batch?","id":"sequential-or-batch"},{"level":2,"text":"Beyond finite domains","id":"beyond-finite-domains"},{"level":2,"text":"Conclusion","id":"conclusion"}]}}