![\"\"](\"https://pncnmnp.github.io/blogs/pics/decision-hierarchy.png\")
Parth Parikh | First published 18 October 2026
\n\n*It is the maxim of every prudent master of a family, never to attempt\nto make at home what it will cost him more to make than to buy. The\ntailor does not attempt to make his own shoes, but buys them of the\nshoemaker. The shoemaker does not attempt to make his own clothes, but\nemploys a tailor. The farmer attempts to make neither the one nor the\nother, but employs those different artificers. All of them find it for\ntheir interest to employ their whole industry in a way in which they\nhave some advantage over their neighbours, and to purchase with a part\nof its produce, or, what is the same thing, with the price of a part\nof it, whatever else they have occasion for.*
\n
- Adam Smith in The Wealth of Nations (1776)
In the past few weeks,\nJev\nand decision models in particular have been the talk of the town. As\nthings are cooling down a bit, I thought it would be fun to\nsystematically explore and forecast where all of this leads to.
\nThe magic, as we all would agree, seems to be in the output, of which\nthere can be three kinds: choice, score and noul. With\nchoice, we provide a fixed list of options and the output is a\nprobability for each option, summing to 1. The score is for\nrating something against an ordered list of levels, using a\nprobability-weighted average of those levels. And finally, the\nnoul,\nwhich some say is named after the Bernoulli trial, returns a number between 0 and 1: the probability that a given\nstatement is true.
\nHere is an example using\nJev 1.13 from OpenRouter:
\nREQUEST: {\n "model": "typesafe/jev-1.13",\n "state": "Some damn good curly fries - seasoning was something.",\n "questions": {"answer": {"type": "score", "instructions": "How enthusiastic is the reviewer?",\n "criteria": ["Lukewarm", "Happy", "Thrilled"]\n}}}\nRESPONSE: {\n "model": "typesafe/jev-1.13-20260917",\n "answers": {"answer": {"type": "score", "score": 1.48, "legend": {"0": "Lukewarm", "1": "Happy", "2": "Thrilled"},\n "probabilities": {"0": 0, "1": 0.51, "2": 0.49},\n "confidence": 0.27}},\n .....\n}There are a couple of interesting consequences as a result of this\nmodel. First, it's a well-defined output! We state the possible\nanswers before asking the question, and the model has to pick among\nthem. Second, every answer carries odds. As we will see later, we\ncan leverage this to do some interesting arithmetic on them. Third, as\nthe expense is in reading the state and consuming the input tokens,\n the output tokens are essentially free. Case in point: I did\nsome napkin math on the above example. Assuming that a regular LLM would\nrequire 250 input tokens and 50 output tokens to produce this response,\nand that we have 400,000 such queries to process, the resulting 100M\ninput tokens and 20M output tokens would cost $13 on GPT-5 nano ($8.5\nwith cached tokens) vs. $4.2 via Jev. And finally, it is fast!
\nPerhaps the most exciting revelation I had, is that this gives rise to a\nnew form of memory hierarchy, which I have labeled as\ndecision hierarchy.
\nAs a refresher, pick your favorite textbook and flip to its memory\nmanagement page. Here is what Tanenbaum has to say about memory\nhierarchies in his\nModern Operating Systems book:
\n\nOver the years, people discovered the concept of a memory hierarchy, in\nwhich computers have a few megabytes of very fast, expensive, volatile\ncache memory, a few gigabytes of medium-speed, medium-priced, volatile\nmain memory, and a few terabytes of slow, cheap, nonvolatile magnetic or\nsolid-state disk storage, not to mention removable storage, such as DVDs\nand USB sticks. It is the job of the operating system to abstract this\nhierarchy into a useful model and then manage the abstraction.
These learnings can be applied in the realm of LLMs too. Assume\nthat you have an L-cache consisting of a mixture of hand-written rules,\nand exact/semantic matching of past answers to the same question and its\nstate. If so, Jev and Decision models can be imagined as the RAM, LLM as\ndisk, and magnetic tapes as human beings. Each level handles what its\nconfident about and passes the misses down to the level below.
\nFigure 1: Memory hierarchy v. decision hierarchy.
\nBut how do we decide on a confidence factor and pick a threshold?\nThis is where\nChow's rule from 1970\ncomes in. Let us motivate the idea first. Imagine a classifier (C)\nwith (n) classes. For each input, the classifier generates a\nprobability for each of the (n) classes. When the largest class has a\nprobability close to 1, the model is sure about its decision, and\nmistakes are rare. However, when all the classes have equal probability,\nsay (1/n), any answer is close to a guess. So, the errors come from\nthe uncertain inputs. And to mitigate it, we need a higher form of\nintelligence - from decision models, to LLMs, and in the worst case\nhumans. As I like to put it, the more “philosophically-NP” (not in a formal sense) a question is, the higher form of\nintelligence it needs to make a non-ambiguous judgement call.
\n*But how do we decide whether we should give a response at a particular\nlevel or abstain from doing so?*\nA simple way would be using threshold (T). For each input, we look at\nthe top probability, say (p_{\\text{max}}). If (p_{\\text{max}} \\leq T), we reject\nand hand the input on to the next level. If (p_{\\text{max}} \\gt T), we answer\nwith the top probability outcome. With this approach, we get a dial that\nmoves from (T = 1/n) to (T = 1). The higher the dial, the more\nrejects we get and fewer errors on what we keep.
\nHowever, this simple threshold approach does not take into consideration\nthe price of the outcome. Particularly,\n*what is the cost of us answering a question wrong at a level, and the\ncost of declining to answer at a level*. Say, a wrong answer costs 1, rejecting costs a fraction (F), and a\nright answer costs 0. If we answer with class (k), we are wrong with\nprobability of (1 - p_k). The probability of being wrong would be at\nits smallest for the top class, (1 - p_{\\text{max}}). Therefore, a second\napproach would be to answer only if we beat the price of rejecting. In\nother words, answer when (1 - p_{\\text{max}} \\lt F).
\nTake an example where (F=0.2) and (p_{\\text{max}}=0.6). As answering costs\n(1 - p_{\\text{max}} = 0.4) on average and rejecting costs 0.2, we reject.
\nNext, we need to evaluate what rejection ((F)) really costs. When we\nare sending a decision to the next level, we need to take into\nconsideration the cost of an error in that level ((\\text{NX}_{\\text{errcost}})), the\ncost we pay to evaluate any input in that level ((\\text{NX}_{\\text{price}})), and\nfinally the error rate on the next level itself ((\\text{NX}_{\\text{err}})).
\nConsider this - say we have 100 software tickets to evaluate and make\ndecisions upon. And we are working at the Jev and LLM layers. Imagine\nthat for all our tickets, Jev is 60% sure about its decision ((p_{\\text{max}}\n= 0.6)). A wrong routing costs $2. The LLM charges $0.01 per ticket\n((\\text{NX}{\\text{price}})) and is wrong 5% of the time ((\\text{NX}{\\text{err}})). The\nLLM-level also incurs a $2 cost ((\\text{NX}_{\\text{errcost}})) when an error is\nproduced at its level. What is the cost incurred in each extreme\nscenario?
\nJev answers all 100 tickets: Jev is wrong on 40 of them. 40 x $2 =
$80.
Send all 100 tickets to LLM: LLM is wrong on 5 tickets. The cost of
LLM evaluation is 100 x $0.01 = $1 in fees. On the 5 wrong tickets,\nthe total cost incurred is 5 x $2 = $10. All in all, $11.
Per ticket, that's $0.80 for answering versus $0.11 for handing on to\nthe next level, so we hand on. Notice that we did not take Jev's fees\ninto account. This is because it's paid in both options, so it can't\nchange which one is cheaper.
\nNow let us derive (F) so that we can apply this in a more general\nscenario.\nRemember that handing on to the next level charges us twice: we pay\nthe fee and absorb the LLM's mistakes.
\nError rate on the next level ((\\text{NX}_{\\text{err}})): 5 wrong answers
per 100 tickets = 0.05 wrong answers per ticket.
How many wrong answers is a fee worth in the next level: $0.01
per ticket / $2 cost for a wrong answer = 0.005 wrong answers. Or\n(\\text{NX}{\\text{price}} / \\text{NX}{\\text{errcost}}).
Therefore, (F = \\text{NX}{\\text{err}} + \\text{NX}{\\text{price}} / \\text{NX}_{\\text{errcost}}).
\nSubstituting it in the original equation, we get Chow's rule: [1
\nFor the above example, (1 - p_{\\text{max}} = 1 - 0.6 = 0.4). And (\\text{NX}_{\\text{err}} +\n\\text{NX}{\\text{price}} / \\text{NX}{\\text{errcost}} = 0.05 + 0.005 = 0.055). As (0.4 \\gt\n0.055), we hand on to the next level, which is the same decision as\n$0.80 versus $0.11.
\nNotice that when errors are expensive, LLM's price barely matters.\nThe only question is who's more accurate on this input. This is\nbecause (\\text{NX}{\\text{price}} / \\text{NX}{\\text{errcost}}) shrinks toward zero, so (F \\sim\n\\text{NX}_{\\text{err}}). In other words, when errors are expensive, a cheaper LLM\nbarely helps, but a more accurate one helps a lot.
\nHowever, when errors are cheap, (\\text{NX}{\\text{price}} / \\text{NX}{\\text{errcost}} \\gt\n\\text{NX}{\\text{err}}). Re-arranging it gives us - (\\text{NX}{\\text{errcost}} \\lt \\text{NX}_{\\text{price}} /\n\\text{NX}_{\\text{err}}). In other words, LLM's fee only matters more than its\nmistakes when a wrong answer costs under a certain amount. For our above\nexample, that's $0.01 / 0.05 or 20 cents. Our tickets cost $2 per wrong\nanswer, well above that, so accuracy decides.
\nTo paraphrase Adam Smith, act only when what we gain from it is more\nthan what we pay in return.\nIn our decision hierarchy, the LLM is selling one product - fewer\nmistakes. Each level is a supplier with 3 numbers: a price, an error\nrate, and the cost of each error it makes. The threshold we discussed\nabove is a break-even point - the point where what the next level\nsaves equals what it charges you.
\nSo far, we have assumed that the decisions made at every level are\nnoiseless. Chow's rule only escalates the inputs that a level is unsure\nabout. However, in the real world, any level is capable of making\nconfident mistakes. This is especially true when two levels share a\nsimilar training. One way to mitigate this is by using a\nrandom audit. Imagine a small random slice, say 1% of Jev's\nconfident decisions, being sent to the next level anyway. This will help\ncalculate a better estimate of the error rate, and tune the costs.
\nChow's rule also assumes that every mistake costs the same. This does\nnot hold up in the real world either. Say, a ticket that needed to be\nsent to the legal team got routed to the tech team instead. Depending on\nthe circumstances, that can cost a lot more than a ticket that was\nsupposed to be sent to the tech team, but got routed to data analytics\ninstead.
\nBefore we manipulate Chow's rule to mitigate this issue, notice how both\n(\\text{NX}{\\text{price}}) and (\\text{NX}{\\text{err}}) don't depend on any output category. As\nper (\\text{NX}_{\\text{price}}), the LLM charges the same fee for every ticket.\nSimilarly, as long as we assume that LLM is equally accurate on every\ncategory, (\\text{NX}{\\text{err}}) doesn't change. So, we only need (\\text{NX}{\\text{errcost}})\nto reflect the output category overhead. Given this, here is the update\nto Chow's rule:
\n((1 - p_{\\text{max}}) \\times \\text{NX}{\\text{errcost}} \\lt \\text{NX}{\\text{err}} \\times \\text{NX}_{\\text{errcost}} +
\\text{NX}{\\text{price}}) (multiply by (\\text{NX}{\\text{errcost}}) on both sides)
Consider (1 - p_{\\text{max}} = \\sum_{c \\neq \\text{max}} p_c) and (1 = \\sum_c
p_c) (where (c) represents a category)
(\\sum_{c \\neq \\text{max}} p_c \\times \\text{NX}{\\text{errcost}} \\lt \\text{NX}{\\text{err}} \\times \\sum_c
p_c \\times \\text{NX}{\\text{errcost}} + \\text{NX}{\\text{price}})
(\\sum_{c \\neq \\text{max}} p_c \\times \\text{NX}{\\text{errcost}}(c) \\lt \\text{NX}{\\text{err}} \\times
\\sum_c p_c \\times \\text{NX}{\\text{errcost}}(c) + \\text{NX}{\\text{price}})
(\\sum_{c \\neq \\text{max}} p_c \\times \\text{NX}{\\text{errcost}}(c) = C' - p{\\text{max}} \\times
\\text{NX}_{\\text{errcost}}(\\text{max}))
(C' - p_{\\text{max}} \\times \\text{NX}{\\text{errcost}}(\\text{max}) \\lt \\text{NX}{\\text{err}} \\times C' +
\\text{NX}_{\\text{price}}) (divide by (C') to put the rule back in the shape of\n the original formula)
(1 - p_{\\text{max}} \\times \\text{NX}{\\text{errcost}}(\\text{max}) / C' \\lt \\text{NX}{\\text{err}} + \\text{NX}_{\\text{price}}
/ C')
Notice how if every category has the same (\\text{NX}_{\\text{errcost}}), then (C' =\n\\text{NX}_{\\text{errcost}}). In which case, we get back our original Chow's rule of\n(1 - p_{\\text{max}} \\lt \\text{NX}{\\text{err}} + \\text{NX}{\\text{price}} / \\text{NX}_{\\text{errcost}}).
\nFigure 2: Interactive plot of the modified Chow's rule (1 - p_{\\text{max}} \\times \\text{NX}{\\text{errcost}}(\\text{max}) / C' \\lt \\text{NX}{\\text{err}} + \\text{NX}_{\\text{price}}\n/ C'). In generic terms, the curve is a hyperbola with the equation (y = 1 - a - b/x), where x is (C'), y is the threshold from the right of the equation, and (a) and (b) are the constants (\\text{NX}{\\text{err}}) and (\\text{NX}{\\text{price}}) respectively. The hyperbola is clamped to prevent negative values. Note that the figure is AI generated using Claude's Opus 5.5 Extra model. In accordance with Anthropic's commercial terms, I own the output, and responsible for its factual correctness, and any copyright issues.
\nTo illustrate the new derivation, let us focus on 3 categories: tech,\nanalytics, and legal. Assume that the LLM still charges us $0.01\n((\\text{NX}{\\text{price}})), is wrong 5% of the time ((\\text{NX}{\\text{err}})), and the output\nprobability on an input is tech 0.6, analytics 0.25, and legal 0.15\nrespectively. Also assume that if the ticket really is tech,\n(\\text{NX}_{\\text{errcost}}) will cost us say $2 (same as before). Similarly, if the\nticket really is analytics or legal, (\\text{NX}_{\\text{errcost}}) will cost us $2\nand $100 respectively.\nAs they say
\n“*I was never ruined but twice - once when I lost a lawsuit, once when
I won one.*”
(C') would be (\\sum_c p_c \\times \\text{NX}_{\\text{errcost}}(c)), or (0.6 × $2 +\n0.25 × $2 + 0.15 × $100), or $16.70. The max category is tech, so\nanswering in tech costs us (C' - p_{\\text{max}} \\times \\text{NX}_{\\text{errcost}}(\\text{Tech})) =\n$16.7 - $1.2 = $15.5. Furthermore, handing on to the next level costs\nus, (\\text{NX}{\\text{err}} \\times C' + \\text{NX}{\\text{price}}) = 0.05 × $16.70 + $0.01 = $0.835
\nAs $15.50 > $0.845, we hand on to the next level. This is the same\nconclusion that we previously made; however, the 15% chance of a legal\nticket makes answering nearly 20 times more expensive (15.5/0.8) than\nbefore.
\nNonetheless, there is a catch with the new derivation. Till now, our\ndecision at every level was based on - should the level answer with its\ntop class or hand on? But\nwith the introduction of a cost table, the top class is not always\nthe cheapest answer. We can have scenarios where the new rule would hand on to the next\nlevel, when answering with a different category would be considered\ncheaper.
\nTo see this, assume that we have the same cost table as before ($2, $2,\n$100), but the probability distribution has changed to tech 0.5,\nanalytics 0.05, and legal 0.45. (C') would be (0.5 × $2 + 0.05 × $2 +\n0.45 × $100), or $46.10. Handing on would cost $0.01 + 0.05 x $46.10 =\n$2.32. Answering with the max costs 0.05 × $2 + 0.45 × $100, or $45.10.\nAnswering with legal costs $1.10. As $1.10 < $2.32, the best decision\nwould be to route to legal. However, with max only - we would have\ndecided to simply hand on to the next level.
\nOne way to deal with this is by letting the math work out as intended -\nthat is, we route to legal, when it's cheaper to do so. But the dilemma\nis that if we route too much, their queue grows, and it ends up turning\ninto a dumping ground for tickets. This would certainly occur if the\ntraining of models at any level is subpar. One way to mitigate such\nissues is by relying on our audit trick. If legal is bouncing\nback lots of tickets or its queue is growing, we raise the price.\nHowever, if real issues are slipping past them, we lower the price.
\n*All of the above analysis does present a threat to Jev and independent\ndecision model-like businesses.*\nParticularly, Jev's advantages are its cheapness, speed, probability\ndistribution, and closed outputs. But those advantages are a direct\nresult of its interface, and interfaces can be copied. If a really cheap\nLLM (say\nGPT-5 nano or Gemini Flash Lite) ends up being more accurate than Jev, then a winning move for vendors\nwould be to call such LLMs in decision mode. Which is exactly what\nOpenAI is doing with their\nDecisions API! The customers get to keep the good stuff: stop paying for the output\ntokens, improved latency, post-training from a strong base model, and\nactual probabilities over the options.\nWhat an interface can't copy is honest probabilities - which is\nwhere Jev's specialized training may hold some edge. As is already the\ncase though, I expect that we should see a great deal of competitors\ncommoditizing Jev's market share and big LLM providers cannibalizing\ntheir own output-token revenue in the coming months to cater to the\nmarket for this interface.
\nIn the future, decisions sold with some form of warranty do not seem\nfar-fetched either. We already have something similar with Riskified\ntoday. As discussed before, when (\\text{NX}_{\\text{price}} / C') goes to zero -\naccuracy alone decides the level at which we answer. At that point, the\neconomic moat solely falls upon the “honest probabilities” and\n“specialized training”. This in itself does not sound appealing.\nHowever, if the independent decision model vendors calibrate per-domain\nand provide a certain guarantee in their contracts, then the needle can\npossibly tilt somewhat in their favor. Imagine the cost being tuned to\naccount for a premium and a deductible, like (\\text{NX}_{\\text{price}} + \\text{premium} +\n\\text{NX}_{\\text{err}} \\times \\text{deductible}).
\nOne topic I have yet to discuss is latency and its addition to Chow's\nmodified rule. One way to do this would be by introducing two variables:\nthe next level's response time in seconds ((\\text{NX}_{\\text{latency}})) and the\ncost per second of wait ((\\text{wait_cost})). With this, the cost of the\nanswer remains the same as before, ((1 - p_{\\text{max}}) \\times\n\\text{NX}_{\\text{errcost}}). However, the cost of handing on to the next level\nbecomes (\\text{NX}{\\text{price}} + \\text{NX}{\\text{err}} \\times \\text{NX}_{\\text{errcost}} + \\text{wait_cost} \\times\n\\text{NX}{\\text{latency}}). This leads to [(1 - p{\\text{max}}) \\times \\text{NX}_{\\text{errcost}} \\lt\n\\text{NX}{\\text{price}} + \\text{NX}{\\text{err}} \\times \\text{NX}_{\\text{errcost}} + \\text{wait_cost} \\times\n\\text{NX}{\\text{latency}}] And dividing by (\\text{NX}{\\text{errcost}}), we get [1 - p_{\\text{max}}\n\\lt \\text{NX}{\\text{err}} + (\\text{NX}{\\text{price}} + \\text{wait_cost} \\times \\text{NX}_{\\text{latency}}) /\n\\text{NX}_{\\text{errcost}}] Just like before, this can similarly be modeled towards\ndifferent output categories too.
\nAs I am writing this, it is fascinating to imagine the possibilities\nthis discussion is leading up to. As I look towards the future, there\nare many questions and opportunities. For instance, how would\nmultimodality interact in the decision model realm? Will we be able to\nsplit the image into grid cells and ask a model to pick a cell given an\ninstruction? Like, which of my cells contains a can of Cherry coke? Or\nwill we be able to count in bins the occurrence of an item - how many\ncars are at the intersection? What about audio? Will we be able to chop\na time window and narrow down to when Jeff Geerling talks about the\nprice of his mini PC? It has been clear for a while now that once such\nmodels cheaply read images, audio, and sensors in real-time, we will be\nable to leverage them inside robots, factory settings, and many\non-device form factors.
\nThe future is exciting and exciting is the future.
","headings":[{"level":1,"text":"To Make, or to Buy: The Economics of Decision Models","id":"to-make-or-to-buy-the-economics-of-decision-models"},{"level":2,"text":"The decision hierarchy","id":"the-decision-hierarchy"},{"level":2,"text":"Answer v. Abstain","id":"answer-v-abstain"},{"level":2,"text":"Deriving Chow's rule","id":"deriving-chow-s-rule"},{"level":2,"text":"What is it telling us?","id":"what-is-it-telling-us"},{"level":2,"text":"Confident mistakes and when errors don't cost the same","id":"confident-mistakes-and-when-errors-don-t-cost-the-same"},{"level":2,"text":"Top class is not always the cheapest","id":"top-class-is-not-always-the-cheapest"},{"level":2,"text":"Bringing out my crystal ball","id":"bringing-out-my-crystal-ball"}]}}