{"article":{"slug":"an-open-field-guide-to-inference-engineering","title":"An Open Field Guide to Inference Engineering","subtitle":null,"summary":"Modelplane's open field guide for platform engineers new to inference, covering how models and GPUs work together, running inference in production, the tooling landscape, and how to grow into an inference engineering role, with figures as of October 2026.","content_type":"tutorial","language":"en","canonical_url":"https://docs.modelplane.ai/guides/inference-guide/","author":{"name":"The Modelplane Authors","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Modelplane","url":"https://modelplane.ai/","listing_slug":null,"listing":null},"topics":[{"name":"ai","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"},{"name":"machine-learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"},{"name":"devops","slug":"devops","url":"https://listedarticles.com/topics/devops"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":8965,"reading_minutes":39,"published_at":"2026-11-10T20:13:44.646Z","added_at":"2026-11-10T20:13:44.646Z","updated_at":"2026-11-10T20:13:44.646Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/an-open-field-guide-to-inference-engineering","markdown_url":"https://listedarticles.com/articles/an-open-field-guide-to-inference-engineering.md","example":false,"citation":"The Modelplane Authors, Modelplane. \"An Open Field Guide to Inference Engineering.\" 10 Nov 2026. https://docs.modelplane.ai/guides/inference-guide/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://docs.modelplane.ai/guides/inference-guide/"},"body_markdown":"# An Open Field Guide to Inference Engineering\n\nA field guide to inference concepts, production operations, tooling, and career development for platform engineers.\n\nSource: https://docs.modelplane.ai/guides/inference-guide/\n\n\n\n*Numbers, prices and tool versions are as of October 2026.*\n\n\nThis guide is for platform engineers new to inference. If you need to understand\nhow models and GPUs work together and how to run inference in production, this\nguide will help. You'll also learn about the tools to use and how to grow into\nan inference engineering role.\n\nThis guide builds on the work of others in the inference community. If you're\ninterested in inference engineering, continue your learning with these:\n\n- Viktor Farcic's [*LLM Inference\n  Explained*](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/)\n- Philip Kiely's [*Inference\n  Engineering*](https://www.baseten.co/inference-engineering/)\n- [The Pragmatic Engineer, What Is Inference\n  Engineering?](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering)\n- [ByteByteGo's Guide to AI Inference\n  Engineering](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering)\n- [Data Gravity on The Rise of Inference\n  Engineering](https://www.datagravity.dev/p/the-rise-of-inference-engineering)\n- [Spheron's Inference Engineering\n  Guide](https://www.spheron.network/blog/inference-engineering-guide-2026/)\n- Vishakha Sadhwani's [*The AI Inference Engineering\n  Roadmap*](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap)\n- Upbound, [*Project\n  Champagne*](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence)\n  series\n\n## 1. Why now?\n\nIn an AI-forward organization, platform teams become inference teams. Coding\nassistants can't send source code to third parties, and frontier model bills\nincrease month to month. Support tools also need internal data that a hosted\nmodel can't reach.\n\nOrganizations want to run these models themselves, and open models now make that\npractical.\n\n[Hugging Face](https://huggingface.co/models) hosts more than two million open\nmodels, and the best of them compete with closed models. DeepSeek released R1 in\nJanuary 2025, and [it matched OpenAI's o1 on math and coding benchmarks about a\nmonth after o1 came out](https://arxiv.org/abs/2501.12948). When you run your\nown model, you control latency, uptime, and where your data goes. At high\nvolume, you can also [cut costs by about\n80%](https://www.baseten.co/inference-engineering/).\n\nThe work suits platform engineers. The price of a fixed level of AI capability\nfalls roughly 10x a year, and most of that comes from how models are served,\n[not from better\nGPUs](https://www.datagravity.dev/p/the-rise-of-inference-engineering). Upbound,\nthe company behind Crossplane and Modelplane, saw this firsthand. In its\n[Project\nChampagne](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence)\nblog series, the team ran its own models on\n[Modelplane](https://github.com/modelplaneai/modelplane) and found that vLLM and\nscheduling worked. The time went to TLS, identity, timeouts, storage, IAM and\nobservability.\n\n### Skills that carry over\n\nAn inference engineer picks a model, gets it onto the right GPU, keeps it\nserving, makes it faster and cheaper, and gets paged when it breaks. That's SRE\nfor a new workload, and most of your skills carry over:\n\n| What you already know        | What it becomes in inference                                                         |\n|------------------------------|--------------------------------------------------------------------------------------|\n| Scheduling and node pools    | GPU placement with device selectors, taints and exclusive devices                    |\n| Capacity planning            | GPU memory math: weights, KV cache and runtime overhead, checked at runtime          |\n| Load balancing               | Cache-aware routing and long-lived streaming connections                             |\n| Autoscaling on CPU           | Scaling on queue depth and KV cache use, with cold starts measured in minutes        |\n| Rollouts                     | Rollouts of exclusive GPUs, and a version matrix of model, hardware and engine build |\n| SLOs and latency percentiles | Time to first token, time per output token and goodput                               |\n| Observability                | Engine metrics, real GPU activity and speculative decoding acceptance                |\n| FinOps                       | Cost per million tokens and cost per user session                                    |\n| Identity and access          | Credentials that last as long as an agent session                                    |\n\nA hosted API is still [the right call early in a product's\nlife](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering).\nSelf-hosting pays off when API spend becomes a real line item, when latency or\nreliability needs outgrow vendors, or when data can't leave your network. In\nAugust 2026, [Chris Zeoli\ncounted](https://www.datagravity.dev/p/the-rise-of-inference-engineering) about\n58 open inference roles on public careers boards, with no settled job title.\n\n## 2. Inference 101\n\nThis section covers the concepts in roughly the order you'll meet them. Most are\nabout what the engine keeps in GPU memory or what limits its speed.\n\n[Viktor\nFarcic](https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/)\nsuggests asking when each concept starts to matter. Some matter as soon as you\ndeploy, others only once many people use the model, and some you may never need.\nThe table near the end of this section gives the signal for each.\n\n### What an inference engine is\n\nAn inference engine, like [vLLM](https://github.com/vllm-project/vllm) or\n[SGLang](https://github.com/sgl-project/sglang), is the server that turns model\nweights into an API. For a platform engineer, the closest mental model is a\nstateful database server with a GPU attached. It loads a large dataset into\nmemory at startup and keeps per-session state. It also schedules work internally\nand answers over HTTP. [Ollama](https://github.com/ollama/ollama) on a laptop is\nthe same idea at small scale, and every hosted model API runs one behind the\nscenes.\n\n| You know this from databases and services         | The engine's counterpart |\n|---------------------------------------------------|--------------------------------------------------------------------|\n| Loading a dataset into the buffer pool at startup | Loading model weights into GPU memory, which can take many minutes |\n| Per-session state in a session store              | The KV cache, one for each conversation in flight                  |\n| A query planner and connection pool               | The scheduler, which batches requests into each GPU pass           |\n| A wire protocol clients already speak             | An OpenAI- or Anthropic-compatible HTTP API                        |\n| Warming caches after a restart                    | Cold starts: pulling a large image and reloading the weights       |\n\nUnlike a database, the engine keeps its working set in GPU memory, not RAM. GPU\nmemory is small and expensive, so most tuning is about deciding how much of it\neach request gets.\n\n### The inference stack\n\nThe engine is one layer of four. At the bottom, **kernels** do the arithmetic.\nYou rarely choose them directly. Above them sits the **engine**. Above that, a\n**control plane** deploys and manages engines, because sooner or later you run\nmore than one. On top, a **gateway** routes traffic across models and replicas.\n\n\n*Inference stack: four layers and the clients they serve*\n\nRequests pass through the gateway to an engine, which runs kernels on the GPU.\nThe control plane deploys and manages the gateway and engines, but no request\npasses through it.\n\n### Tokens\n\nA token is the unit a language model works in, roughly a word or word fragment.\nContext windows, prices and speeds are all counted in tokens. When someone\nquotes tokens per second, they mean this unit.\n\n### Weights and the KV cache\n\nThe weights are fixed, and the KV cache grows with load. Size for both, as you\nwould a database's footprint and working set.\n\n**Weights are the fixed footprint.** Most models are released at 2 bytes per\nparameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live\nin GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of\nthe card for everything else. That number is the same for one user or a hundred.\n\n**The KV cache is the working set.** Each new token is computed against every\ntoken before it. Rather than recompute that history, the engine keeps the\nattention keys and values for every token in flight. Its size scales with\ncontext length multiplied by the number of concurrent conversations, much like a\nsession store that grows with both session size and user count.\n\nYou can estimate the KV cache per token from a model's config:\n\n```text\nKV bytes per token = 2 × layers × KV heads × head dimension × bytes per value\n```\n\nFor Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is\nabout 320 KB per token. One 128K-token conversation needs about 40 GB of cache.\nAt that size, the KV cache usually limits how many users a GPU can serve.\n\nWatch for these sizing mistakes:\n\n- **Mixture-of-experts (MoE) models keep every weight resident.** A model with\n  320B total parameters and 18B active still needs about 320 GB at FP8. Only\n  compute scales with the active parameters.\n\n- **Read the checkpoint size, not the dtype table.** While self-hosting its own\n  models for [Project\n  Champagne](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks),\n  Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The\n  real checkpoint was 177.7 GB, because the experts were stored in a 4-bit\n  format.\n\n- **Leave room for runtime overhead.** CUDA graphs, activations and memory\n  fragmentation all take space. A model that fits at startup can still run out\n  of memory days later ([Project Champagne, Part\n  2](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks)).\n\n### Two phases, two bottlenecks\n\nEach request runs in two phases on the same GPU, and [they are limited by\ndifferent\nthings](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering).\n\n| Phase   | What it does                                                                                                                                                                                             | Limited by       | Metric it sets                              |\n|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|---------------------------------------------|\n| Prefill | Reads the whole input in parallel, including the system prompt and tool definitions as well as the conversation history. Produces the first token and the KV cache.                                                                     | Compute          | Time to first token (TTFT)                  |\n| Decode  | Writes the answer one token at a time. Each token needs a pass that reads all active weights, plus the KV cache, from memory. At long context, reading the cache can cost more than reading the weights. | Memory bandwidth | Time per output token and tokens per second |\n\nGenerating one token does one or two floating-point operations per byte read\nfrom memory, depending on weight precision. An H100 needs about 295 operations\nper byte to stay busy. That means one request uses a small fraction of the chip\nwhile the compute units wait on memory ([Data\nGravity](https://www.datagravity.dev/p/the-rise-of-inference-engineering)).\n\nPeak compute has grown about 3x every two years, against about 1.6x for memory\nbandwidth. Nearly every optimization either moves fewer bytes per token or\nspreads each byte across more tokens.\n\nBecause the phases have opposite bottlenecks, a technique that speeds one often\ndoes little for the other. That is why benchmarks report TTFT and tokens per\nsecond separately.\n\n### Metrics to track\n\n| Metric                                        | What it measures                                                       | Watch out for                                                                                                  |\n|-----------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|\n| Time to first token (TTFT)                    | Queueing plus prefill: how long the user waits before anything appears | With reasoning models, the engine's first token may be hidden thinking. Measure the first token the user sees. |\n| Time per output token, or inter-token latency | Decode speed per stream                                                | Speculative decoding and memory bandwidth show up here                                                         |\n| Tokens per second per user                    | Interactivity, or the model's typing speed                             | Falls as batch size rises                                                                                      |\n| Throughput                                    | Total tokens per second across all users                               | Rises as batch size rises                                                                                      |\n| Goodput                                       | Throughput that still meets your latency targets                       | The number to plan capacity with                                                                               |\n| End-to-end latency                            | Total request time                                                     | Depends on how much the model writes. A healthy 190-second answer is normal for long generations.              |\n| Percentiles                                   | p50, p90, p99 of each latency                                          | Users feel the tail, not the average                                                                           |\n| Errors and refusals                           | Failed or rejected requests                                            | Count refusals as errors, or a context cap can look like a speedup                                             |\n| Cost per million tokens                       | GPU cost divided by tokens served                                      | Compare at the load you actually run                                                                           |\n\nCost per million tokens converts a GPU price and a throughput into one\ncomparable number:\n\n```text\nCost per million tokens = GPU cost per hour / (tokens per second × 3600) × 1,000,000\n```\n\n[Spheron's](https://www.spheron.network/blog/inference-engineering-guide-2026/)\nworked example: a GPU at \\$2.90 an hour sustaining 1,200 tokens per second costs\nabout \\$0.67 per million tokens.\n\n### Latency versus throughput\n\nBatching more requests together raises total throughput and slows each\nindividual user. Interactive chat and coding favor latency. Overnight batch jobs\nfavor throughput.\n\nIt's the same trade-off as tuning a database for transactions or for analytics.\nWrite down the workload's constraints before you tune anything.\n\n### When to adopt each concept\n\nEach row gives the symptom that tells you it's time to act, in roughly the order\nteams need them.\n\n| Concept                                | What it does                                                            | The signal that you need it                                                                                                       |\n|----------------------------------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|\n| Fitting the weights                    | Choose a model, precision and GPU so the weights fit with room to spare | The engine fails to start, or starts with almost no KV cache left                                                                 |\n| Continuous batching and paged KV cache | Share each GPU pass across requests, and hand out cache in small blocks | Already on by default in vLLM and SGLang. Check it, don't build it.                                                               |\n| Quantization                           | Store weights in fewer bits, and give the freed memory to the KV cache  | Requests queue while the GPU still has compute to spare                                                                           |\n| Prefix caching                         | Reuse the work already done on repeated prompt openings                 | Already on by default in vLLM and SGLang. Watch the hit rate once your clients are agents or chat apps that resend long histories |\n| Speculative decoding                   | Draft several tokens and verify them in one pass                        | Individual responses stream too slowly, even at low load                                                                          |\n| Gateway and control plane              | Route between replicas, and manage engines declaratively                | You add a second model, a second replica or a second cluster                                                                      |\n| Sharding across GPUs                   | Split one model with tensor, pipeline or expert parallelism             | The model will not fit on one GPU. Plan this before you deploy.                                                                   |\n| Autoscaling                            | Add and remove replicas on queue depth and KV cache use                 | Idle GPUs show up on the bill, or queues build at peak hours                                                                      |\n| Disaggregation                         | Run prefill and decode on separate GPUs                                 | Long prompts stall streaming at scale, and you have RDMA-class networking                                                         |\n\nMost teams need only the first few rows on day one. Add the rest when their\nsignals show up.\n\n### What agents change\n\nCoding agents and other agentic workloads stress every part of this picture.\nUpbound measured this in [Part\n5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team)\nof its Project Champagne series (introduced in Section 1), where about 10 of its\nengineers ran Claude Code and other coding agents against a self-hosted model.\nTheir traffic looked like this:\n\n- **Prompts are long.** The median prompt was 59K tokens and the p90 was 125K.\n\n- **Prompts are mostly repeated.** About two-thirds of prompt tokens hit the\n  prefix cache, because agents resend their transcript on every turn.\n\n- **Thinking is hidden.** A reasoning model can think for more than a minute\n  before its first visible token. Every timeout between the agent and the model\n  has to allow for that.\n\n- **Load depends on the client.** Short-prompt sessions launch subagents and can\n  keep more requests in flight than long-prompt sessions. Plan capacity in\n  requests in flight rather than seats.\n\n- **Sessions run for hours.** Streams, tool calls and credentials all have to\n  keep working for that long.\n\n## 3. Running inference in production\n\nIn production, every choice trades latency, throughput, cost and quality against\neach other, usually with too few GPUs. Where a practice comes from Project\nChampagne, Section 5 has the details.\n\n### 3.1 Start from use cases, not hardware\n\nEach use case sets different requirements. In Project Champagne:\n\n- **Coding agents** needed long context without quality loss, reliable tool-call\n  parsing, streaming, sessions that last hours, and an Anthropic-compatible API\n  for Claude Code.\n\n- **Chat** needed low time to first token, real concurrency, and an\n  OpenAI-compatible API for existing tools.\n\n- **The platform** needed TLS on a real hostname and identity tied to the\n  company domain. It also needed observability that tells a healthy engine from\n  a busy one, and correct placement without babysitting.\n\nSeparate interactive workloads from batch workloads. They need different\ntargets, and often different deployments. Write the pass and fail bar down\nbefore you test anything.\n\n### 3.2 Pick models deliberately\n\n- **Evaluate on your own tasks.** Public benchmarks measure someone else's\n  workload. Section 3.11 covers how.\n\n- **Start small, then grow.** A small MoE model with a few billion active\n  parameters builds operating experience faster and costs less to get wrong.\n\n- **The newest model is not always the right one.** Model releases move faster\n  than hardware plans. A new ‘small’ flagship can need several of the largest\n  cards, while your current model may be running at half its supported context.\n  The cheapest gain may not need a new model at all.\n\n- **Check that your engine can actually load it.** A model listed as supported\n  in release notes may still fail to load on your hardware and engine build.\n\n- **Check the license.** Most open models use permissive licenses, but some\n  restrict commercial use.\n\n- **Check the chat template and parsers.** Tool calls and reasoning output\n  depend on settings each model needs in the engine. A wrong one breaks tool use\n  while every request still returns 200.\n\n### 3.3 Size for runtime, not startup\n\nA model that fits at startup can still run out of memory under real traffic when\nfragmentation pushes past the profiled peak. That can happen days later, in the\nmiddle of a request.\n\n- **Budget all three costs:** weights, KV cache, and runtime overhead such as\n  CUDA graphs and activations.\n\n- **Know what your engine's memory setting covers, and check again on every\n  upgrade.** vLLM sizes its KV cache from a startup estimate of everything else,\n  including CUDA graph pools, and real traffic can go past that estimate. If you\n  see out-of-memory errors under load, lower the memory fraction slightly,\n  enable expandable segments and cap concurrent sequences, then measure free\n  memory again.\n\n- **Cap every request.** Set a maximum output length on every call. Without one,\n  any request can take a model down for everyone, and abandoned requests keep\n  generating after the client gives up.\n\n- **Shorten context with care.** A shorter maximum context frees KV cache for\n  more users. It also rejects every longer conversation, so count those\n  rejections as errors.\n\n### 3.4 Know the optimization techniques\n\nMost engines turn on continuous batching and paged KV cache by default. The rest\nare choices you make per model and workload.\n\n| Technique                                                                               | What it improves                                                                                                                                                                                                                   | Cost or risk                                                                                          | Use it when                                                                                                                      |\n|-----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|\n| Continuous batching                                                                     | Throughput. [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) reported 36.9x over a static baseline at the same latency.                                                                                     | Per-user speed drops as batches grow                                                                  | Always. It is the default in modern engines.                                                                                     |\n| Paged KV cache ([PagedAttention](https://arxiv.org/abs/2309.06180))              | Concurrency. vLLM cut KV cache waste from 60 to 80% to under 4%.                                                                                                                                                                   | Little in practice                                                                                    | Always. It is the default in vLLM.                                                                                               |\n| Prefix caching                                                                          | TTFT and prefill cost on repeated openings                                                                                                                                                                                         | Only helps up to the first token that differs                                                         | System prompts, RAG, multi-turn chat and agents                                                                                  |\n| Weight quantization                                                                     | Decode always, and prefill only when activations are quantized too, as with FP8. Weight-only formats such as AWQ and GPTQ speed up decode alone. One step down in precision is typically 30 to 50% faster (Inference Engineering). | Quality loss, which you must measure                                                                  | Use the lowest precision your hardware runs natively and your evals accept. FP8 runs natively on Ada, Hopper and Blackwell GPUs. |\n| KV cache quantization                                                                   | More cache fits, and it moves faster                                                                                                                                                                                               | Errors compound from token to token                                                                   | When KV cache is the limit                                                                                                       |\n| [Speculative decoding](https://arxiv.org/abs/2211.17192), multi-token prediction | Tokens per second at low batch sizes                                                                                                                                                                                               | Wasted compute when guesses are rejected. Its value shrinks as load rises and spare compute runs out. | Latency matters and output is predictable                                                                                        |\n| [Tensor parallelism](https://arxiv.org/abs/1909.08053)                           | Fits large models, faster per user                                                                                                                                                                                                 | Needs fast GPU-to-GPU links such as NVLink                                                            | Large models within one node                                                                                                     |\n| Expert parallelism                                                                      | Throughput for [MoE](https://arxiv.org/abs/1701.06538) models                                                                                                                                                               | Routing between experts                                                                               | MoE models, including across nodes                                                                                               |\n| Pipeline parallelism                                                                    | Fits a model across nodes on ordinary networking                                                                                                                                                                                   | Adds latency to each pass                                                                             | The model does not fit in one node                                                                                               |\n| Prefill and decode disaggregation                                                       | Removes interference between phases. [DistServe](https://arxiv.org/abs/2401.09670) reported 7.4x more requests within latency limits.                                                                                       | Needs RDMA-class networking to move the KV cache                                                      | Large scale, long prompts, fast interconnect                                                                                     |\n\nA few rules of thumb:\n\n- **Structure prompts for the cache.** Prefix caching stops at the first token\n  that differs. Put shared content such as system prompts and tool definitions\n  first, and variable user input last. [Hosted APIs price cached input at 10 to\n  50% of fresh input, as of 2026, for the same\n  reason](https://www.datagravity.dev/p/the-rise-of-inference-engineering)).\n\n- **Quantize from least to most sensitive.** Linear weights tolerate\n  quantization best, then activations, then the KV cache. Attention layers are\n  the most sensitive, because small errors compound across a long sequence. Keep\n  the most sensitive parts at higher precision until your evals show you can go\n  lower. Floating-point formats such as FP8 handle outlier values better than\n  integer formats.\n\n- **Watch speculative decoding acceptance.** Acceptance depends on temperature,\n  subject matter and the draft model. If acceptance collapses, every pass still\n  pays for the guesses and the model slows while other panels look normal.\n\n- **Beware interactions when applying multiple techniques.** Quantizing the KV\n  cache helps disaggregation. Larger batches leave less spare compute for\n  speculation. [One Baseten engineer tried 77\n  configurations](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering)\n  before finding one that doubled throughput for a customer (Inference\n  Engineering). Change one thing at a time and measure (Section 3.10).\n\n- **Keep long prompts from stalling other streams.** Chunked prefill splits a\n  long prompt across several passes, so decode keeps moving for everyone else.\n  Current vLLM turns it on by default, and its token budget per pass is the\n  setting to tune before you consider disaggregation.\n\n### 3.5 Get GPUs, and place them correctly\n\n- **Quota doesn't guarantee capacity.** Quota only lets you request instances.\n  Each instance size is its own capacity pool, even with the same GPU, and no\n  cloud API tells you what you can actually launch.\n\n- **Design for whatever GPUs you can get.** Expect to get the GPU you planned\n  for only some of the time. A model may end up split across smaller nodes in\n  different zones, using pipeline parallelism over ordinary networking. Allow\n  multi-zone pools, and keep the model definition independent of the instance\n  type and node count.\n\n- **Match models to hardware explicitly.** Kubernetes Dynamic Resource\n  Allocation (DRA) device selectors can pin a model to a GPU architecture or a\n  minimum memory size. Verify placement against what the GPU driver publishes,\n  not against a status field that echoes what you declared.\n\n- **Watch the cloud, not just the controller.** A managed node group can report\n  failure while the cloud keeps trying to launch instances, and GPUs that arrive\n  hours later can be shut down by a later change to the pool. Track instances\n  and node joins, and list a pool's instances before you change it.\n\n- **Know your interconnect.** SXM datacenter GPUs such as H100, H200 and B200\n  link GPUs with NVLink. PCIe cards such as the L40S and RTX PRO 6000 do not, so\n  multi-GPU communication goes over PCIe and host memory. That changes which\n  parallelism strategies pay off.\n\n- **Size the system pool too.** GPU nodes are tainted for inference. Monitoring,\n  controllers and gateways compete for whatever general capacity is left.\n\n### 3.6 Gateways, routing and timeouts\n\n- **Do not round-robin.** One replica may already have a conversation's prefix\n  in cache while another has never seen it. Use cache-aware or prefix-aware\n  routing, such as the endpoint picker in the [Gateway API Inference\n  Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension).\n\n- **Route by model name.** One hostname can serve many models, routed on the\n  request's model field.\n\n- **Speak both common APIs.** Most tools expect the OpenAI API. Claude Code\n  expects the Anthropic Messages API, which\n  [vLLM](https://github.com/vllm-project/vllm) serves natively, including\n  streaming and tool calls.\n\n- **Raise every timeout on the path.** Long generations and reasoning models\n  break the defaults, because each hop has a default timeout sized for web\n  requests, not for generations that run for minutes. Typical culprits are a\n  gateway route timeout that bounds the whole response, a load balancer idle\n  timeout, and retries that multiply the wait. Check the route, each load\n  balancer, the gateway, the client and any helper calls the client makes.\n\n- **Detect truncated streams.** A cut stream can return HTTP 200 with valid\n  tokens and then stop. To the client, it looks like a finished answer.\n\n- **Meter usage per caller.** Token-based rate limits and quotas at the gateway\n  keep one heavy user or runaway agent from starving everyone else.\n\n### 3.7 Security and identity\n\n- **Never expose an open endpoint.** Apply your auth policy before the gateway\n  gets an address, and remove it last when you tear down.\n\n- **Use real TLS on a real hostname,** with a plan for certificate issuance and\n  renewal.\n\n- **Plan for three kinds of client.** People are the simplest case. Browsers\n  cannot attach a bearer token to page loads or preflight requests, so handle\n  browser login with OIDC at the gateway. Agents run for hours and read their\n  credential once, so a 60-minute identity token will end them mid-session. Give\n  agents long-lived API keys or a credential that refreshes.\n\n- **Do not stack two expiring sessions.** An outer gateway login in front of an\n  app with its own login causes an outage when the outer session expires.\n\n- **Test what should be refused.** Build a probe matrix of requests the gateway\n  must reject, such as a missing, garbage, unsigned, expired or revoked key, or\n  a forged caller header. Include paths that must remain reachable, such as the\n  health check. Run it on a schedule and in a loop during changes.\n\n- **Carry identity into usage records,** so you can see who sends what.\n  Overwrite any caller header a client tries to forge.\n\n- Treat model weights like code. Prefer safetensors over pickle-based\n  checkpoints, don't enable remote code from a model repo you haven't read, and\n  pin each model to a specific revision.\n\n### 3.8 Observability\n\n- **Scrape the engines.** Engines expose detailed Prometheus metrics. Nothing\n  collects them unless something is set up to scrape them, such as a\n  ServiceMonitor and a named port.\n\n- **Build the core panels:** requests running against the concurrency limit,\n  requests waiting, KV cache use, TTFT, time per output token, output tokens per\n  request, prefix cache hit rate, speculative decoding acceptance, errors by\n  status code, and cancellations.\n\n- Measure quality on the server. Broken tool calls, cut-off answers and empty\n  responses usually reach you as user complaints while the serving dashboards\n  look normal. Count them on the server instead, including tool calls that fail\n  to parse, responses cut off by the length limit, empty outputs, refusals and\n  hidden reasoning tokens per request. The gateway sees every response, so it\n  can count anything the engine doesn't export.\n\n- **Look past `GPU-Util`.** It reports whether anything ran, not how much of the\n  chip worked. A GPU can report near 100% `GPU-Util` while SM activity is around\n  half and tensor cores sit mostly idle. Graph SM activity and tensor activity\n  as well.\n\n- **Tell the kinds of slow apart.** When requests are queued, running is at the\n  limit, waiting is above zero and the KV cache is full. When the model is\n  thinking, TTFT looks fine but each request writes many hidden tokens. When\n  generation is slow, time per output token is up.\n\n- **Watch the watcher.** Something outside the monitoring stack must check that\n  data is fresh and that new metrics arrive from every cluster. A metrics\n  adapter can report its API as available while returning empty results.\n\n- **Keep logs you can debug from.** Metric scrapes can make up nearly all of a\n  retained engine log. Keep per-request access logs with the caller, send them\n  to a log store, and remember `kubectl logs --previous` after a crash.\n\n- **Normalize metric names across engines,** so a vLLM deployment and an SGLang\n  deployment are judged on the same signals.\n\nSet SLOs on what users feel, measured per request at the front door:\n\n- **Responsive:** time to the first token the user actually sees, under a\n  threshold.\n\n- **Fast enough:** time per output token, under a threshold.\n\n- **Answered:** the request completed without an error, and any tool call in it\n  parsed.\n\nDo not set an SLO on total request time, because it depends on how much the\nmodel writes. Low-traffic models need a fleet-wide SLO, longer windows or\nsynthetic traffic. Missing data reads as 100% good, so keep a separate alert for\nmetrics that stop arriving. Open source tools such as\n[Sloth](https://github.com/slok/sloth) and\n[Pyrra](https://github.com/pyrra-dev/pyrra) turn SLO definitions into burn-rate\nalerts. For background, see [*Observability\nEngineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/).\n\n### 3.9 Autoscaling and cold starts\n\n- **Scale on inference signals,** such as queue depth, KV cache use and requests\n  in flight. CPU usage tells you almost nothing here.\n\n- **Combine traffic and load.** Traffic lets you scale ahead of demand, and load\n  lags behind it. The settings are the ones you already tune on any autoscaler.\n  Set minimum and maximum replicas and the length of the metrics window. Then\n  set the scale-down delay and how many concurrent requests one replica should\n  hold. [Kiely's book](https://www.baseten.co/inference-engineering/) covers\n  each in depth for inference.\n\n- **Plan for slow cold starts.** A new GPU node must boot, pull a large engine\n  image and load the weights. Loading weights alone can take half an hour for a\n  large model. Cache weights on shared storage in each cluster so new replicas\n  do not download them again.\n\n- **Be careful with scale-to-zero.** By the time a node is ready, the spike may\n  be over, and scarce GPU capacity may not come back.\n\n- **Plan for multiple clusters at scale.** Past a few hundred GPUs, capacity\n  pushes workloads across regions and providers. A control plane that treats\n  separate pools as one avoids leaving capacity idle in one pool while another\n  runs short.\n\n### 3.10 Change management\n\n- **Track a three-way version matrix:** model, hardware and engine build. New\n  models often need a nightly or model-specific engine build on day zero.\n\n- **Plan rollouts for exclusive GPUs.** A GPU claim is exclusive, so a rolling\n  update on a one-replica deployment deadlocks: the new pod cannot schedule and\n  the old one never leaves. Use a Recreate strategy or allow one unavailable,\n  and budget the downtime.\n\n- **Ready doesn't mean routable.** A gateway can lag a ready replica by a minute\n  or two.\n\n- **Test serving changes like code.** Make each change a pull request and change\n  one thing at a time. Write the bar down first. Sweep load upward until a step\n  misses the bar, and report capacity as one number with a confidence interval.\n\n- **Test the test rig first.** A rig bug can waste a scarce GPU window that a\n  few minutes on a laptop would have caught.\n\n- **Test on your own hardware with traffic like yours.** Agent prompts are long\n  and mostly cached. A benchmark of short, fresh prompts measures a different\n  system.\n\nPromote every serving change through tiers. Each tier costs more than the one\nbefore and catches a different kind of failure, so most problems show up before\nusers see them:\n\n- **Offline.** Validate the config and run the test rig's own checks. This takes\n  minutes on a laptop and no GPU.\n\n- **Rehearsal.** Run the full test end to end against a fake gateway or a small\n  model, so rig bugs don't cost a GPU window.\n\n- **Smoke.** Send a few real requests on the target hardware. Include a long\n  prompt, a tool call and a streamed answer, plus one request that should be\n  refused.\n\n- **Load with evals.** Sweep concurrency up to your bar with the eval suite\n  running alongside, and compare against the current version on the same\n  traffic. Evals can pass one request at a time while tool calls break under\n  concurrent load, with nothing moving on the serving dashboards.\n\n- **Canary.** Send a small share of real traffic to the new version and compare\n  its server-side signals with the current one: TTFT, time per output token,\n  errors, tool-call parse failures and length-limit stops. A gateway that routes\n  by model name can split traffic by weight. Write the rollback trigger down\n  before you start.\n\n- **Full rollout.** Keep the old version ready to take traffic back until the\n  new one's signals stay within the bar through a peak in traffic.\n\n### 3.11 Quality and evals\n\nWhen an open model scores low, check three places before you blame the model:\n\n1.  **The harness.** Is the model the only thing that changed? Run your current\n    hosted model first. If it scores below a weaker model, the harness is\n    broken.\n\n2.  **The infrastructure.** Did the model get a chance to answer? Timeouts,\n    expired credentials and routing errors show up as model failures. Mark those\n    runs \"not measured.\"\n\n3.  **The grader.** Does the check measure the outcome you care about? Grade the\n    end state, not the agent's description of it, and test your graders against\n    answers people already accepted or rejected.\n\nAlso pin the data the right answer depends on and give every model the same\ncontext. Keep test markers where the model cannot read them. Run each test more\nthan once, and prefer deterministic checks over using a model as the judge. A\nbatching server can return different tokens for the same request even at\ntemperature 0, so expect some variance unless the engine runs in a slower\nbatch-invariant mode.\n\n### 3.12 Incidents and debugging\n\n- **Expect silent failures.** The costly problems often don't throw errors: a\n  truncated stream returns 200, a metrics API reports available, node groups\n  report ready. Part of the platform’s job is to make those failures loud.\n\n- **Look at the duration.** A failure at a round number of seconds is a timeout.\n  A multiple of it, such as 239 seconds after four 60-second tries, points to\n  retries.\n\n- **Bypass a layer.** Take one hop out of the path and see whether the problem\n  moves.\n\n- **Verify status against reality.** A status is one component's view of its own\n  job. Check the disk, the instance or the route itself.\n\n- **Work additively, and upstream the fix.** Keep workarounds beside the\n  platform, labelled for teardown, rather than editing objects a controller\n  owns.\n\n### Checklist before your first users\n\n- [ ] Use cases defined, with a written bar for time to first token, speed per\n  user and error rate\n\n- [ ] Model chosen on your own evals, with your current hosted model as the\n  baseline\n\n- [ ] Engine build confirmed to load the model on your hardware\n\n- [ ] Memory budget covers weights, KV cache and runtime overhead, with max\n  output tokens and max concurrent sequences set\n\n- [ ] GPU placement verified against the driver, with a fallback instance shape\n  or zone\n\n- [ ] Auth policy applied before the gateway gets an address, and TLS on a real\n  hostname\n\n- [ ] Credentials that last as long as an agent session\n\n- [ ] Timeouts raised at every hop: route, load balancers, gateway retries,\n  client and helper calls\n\n- [ ] Engine metrics scraped, GPU SM and tensor activity graphed, and an alert\n  when metrics stop arriving\n\n- [ ] Server-side counts of tool-call parse failures, length-limit stops and\n  empty responses, so users aren't your first alert\n\n- [ ] A probe matrix of requests that must be refused and paths that must stay\n  open\n\n- [ ] A rollout strategy for exclusive GPUs, with downtime budgeted\n\n- [ ] Previous container logs preserved after a crash\n\n- [ ] Serving changes promoted through tiers, with evals under concurrent load\n  and a canary with a written rollback trigger\n\n- [ ] Workarounds kept beside the platform, labelled for teardown, with fixes\n  sent upstream\n\n## 4. The stack\n\nGo deep on the engine. You already know Kubernetes, gateways and observability,\nso learn what changes for inference. For kernels, reading a profile is enough.\nMost inference engineering happens higher up the stack.\n\n### Tools by layer\n\n| Layer                  | Tools to know                                                                                                                                                                                                                                                                                                                                  | Master first                                                                                                                                |\n|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|\n| GPUs on Kubernetes     | [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator), [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) and its [NVIDIA driver](https://github.com/NVIDIA/k8s-dra-driver-gpu), [DCGM exporter](https://github.com/NVIDIA/dcgm-exporter) | How GPUs are advertised, requested and placed                                                                                               |\n| Kernels and libraries  | [CUDA](https://developer.nvidia.com/cuda-toolkit), [PyTorch](https://pytorch.org), [Triton](https://github.com/triton-lang/triton), [FlashAttention](https://github.com/Dao-AILab/flash-attention), torch.compile                                                                                                  | Kernel basics, kernel fusion, and how to read a profile                                                                                     |\n| Inference engines      | [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM); [Ollama](https://github.com/ollama/ollama) and [llama.cpp](https://github.com/ggml-org/llama.cpp) for local work                                     | vLLM's core flags: `--gpu-memory-utilization`, `--max-num-seqs`, `--max-model-len`, tensor parallel size, quantization, prefix caching, chunked prefill |\n| Distributed serving    | [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [llm-d](https://github.com/llm-d/llm-d), [KServe](https://github.com/kserve/kserve), [LeaderWorkerSet](https://github.com/kubernetes-sigs/lws)                                                                                                               | Multi-GPU and multi-node layouts, and prefill and decode split                                                                              |\n| Gateway and routing    | [Envoy Gateway](https://github.com/envoyproxy/gateway), [Agent Router](https://theagentrouter.ai) (formerly Envoy AI Gateway), [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)                                                                                      | Streaming, timeouts, auth policies, model-name and cache-aware routing                                                                      |\n| Control plane          | [Modelplane](https://github.com/modelplaneai/modelplane), built on [Crossplane](https://github.com/crossplane/crossplane)                                                                                                                                                                                                        | Declaring clusters, hardware classes, deployments and caches across environments                                                            |\n| Observability          | [Prometheus](https://prometheus.io), [Grafana](https://github.com/grafana/grafana), [OpenTelemetry](https://opentelemetry.io), [Loki](https://github.com/grafana/loki), [Sloth](https://github.com/slok/sloth) or [Pyrra](https://github.com/pyrra-dev/pyrra)                                        | Engine metrics, real GPU activity, and SLOs on user-facing latency                                                                          |\n| Benchmarking           | [vLLM's benchmark tools](https://github.com/vllm-project/vllm), [NVIDIA AIPerf](https://github.com/ai-dynamo/aiperf), [GuideLLM](https://github.com/vllm-project/guidellm), [SemiAnalysis AgentX](https://inferencex.semianalysis.com/agentx/methodology) traces                                                   | Concurrency sweeps and latency percentiles under realistic prompts                                                                          |\n| Evals                  | [Inspect](https://inspect.aisi.org.uk) from the UK AI Security Institute; Hugging Face upskill for testing skills across models                                                                                                                                                                                                         | Deterministic graders on the end state of a task                                                                                            |\n| Models                 | [Hugging Face Hub](https://huggingface.co/models), [safetensors](https://github.com/huggingface/safetensors) checkpoints                                                                                                                                                                                                         | Reading a model config for layers, heads and context, and checking real file size                                                           |\n\n### Choosing an engine\n\nThe engine controls batching, memory management, kernel selection and API\ncompatibility. It often matters more to cost and throughput than a hardware\nupgrade.\n\n| Engine                                                                                                       | Why teams pick it                                                                                                                                                                                                         | What it costs to operate                                                                   |\n|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|\n| [vLLM](https://github.com/vllm-project/vllm)                                                          | The broadest model coverage, plus OpenAI and Anthropic APIs out of the box, so it is the usual first deployment. Hosted by the PyTorch Foundation.                                                                        | Frequent releases with new flags. Pin versions and re-benchmark on every upgrade.          |\n| [SGLang](https://github.com/sgl-project/sglang)                                                       | A radix-tree prefix cache and a scheduler that favors requests sharing a cached prefix, which suits agents and long multi-turn sessions. Upbound measured its biggest single gain here (Project Champagne, Part 5, 2026). | A smaller community, and new models may need nightly builds before a tagged release works. |\n| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)                                                | Kernels tuned for NVIDIA hardware, including FP4 on Blackwell                                                                                                                                                             | NVIDIA hardware only, which ties your serving stack to one vendor.                         |\n| [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo)                                                  | A layer above engines that coordinates many nodes, including split prefill and decode                                                                                                                                     | Another distributed system to run, so use it only at multi-node scale.                      |\n| [Ollama](https://github.com/ollama/ollama), [llama.cpp](https://github.com/ggml-org/llama.cpp) | Local serving for learning and prototyping                                                                                                                                                                           | Not designed for many concurrent users                                                     |\n\nHugging Face's TGI was a common choice and is now in maintenance mode, so check\nrelease cadence, maintainers and foundation backing before you commit to one.\nEngines add new flags every few weeks. Re-run your benchmarks when you upgrade,\nrather than trusting last quarter's numbers.\n\nUpbound's own comparison in [Project Champagne, Part\n5](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team)\nshows this. The team was serving a coding model to about 10 engineers on two\n8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the\nsame agent traffic, switching from vLLM to SGLang was the largest single gain it\nmeasured. Adding SGLang's host-memory cache on top didn't add a measurable gain,\nbecause the KV cache never filled.\n\n### Where Modelplane fits\n\n[Modelplane](https://github.com/modelplaneai/modelplane) is an open source\nproject from Upbound, built on\n[Crossplane](https://github.com/crossplane/crossplane) and released under the\nApache 2.0 license. It orchestrates models, inference engines and AI\ninfrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style\ndeclarative APIs. It provisions inference clusters or uses ones you bring.\nDocumentation is at [docs.modelplane.ai](https://docs.modelplane.ai).\n\nModelplane splits the work into three layers of ownership. You can use the same\nsplit as a mental model even with different tools.\n\n| Who                                   | Owns                                                  | Modelplane resources                               |\n|---------------------------------------|-------------------------------------------------------|----------------------------------------------------|\n| Platform team                         | The fleet: clusters, hardware classes and the gateway | InferenceCluster, InferenceClass, InferenceGateway |\n| Model operator, or inference engineer | The models: deployments, services and weight caches   | ModelDeployment, ModelService, ModelCache          |\n| Modelplane                            | Everything in between, composed from the two above    | ModelEndpoint, ModelReplica                        |\n\nThe hierarchy mirrors core Kubernetes one scope up. ModelDeployment →\nModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service\n→ Endpoint, across a fleet instead of within one cluster. You bring the engine\ncontainer and its flags, and Modelplane shapes the topology around them. People\nand agents never create any of these. They call the InferenceGateway by model\nname.\n\nModelplane maps vLLM and SGLang metrics onto common names out of the box, and a\nMetricMapping adds any other engine, so different engines are judged on the same\nsignals.\n\nFor a platform engineer, the second layer is where the new learning is. Upbound\nfound the same thing when it started self-hosting in [Project Champagne, Part\n1](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence):\nthe cluster and gateway work was familiar, and the model operator layer held the\nunknowns.\n\n### A suggested order to learn the stack\n\n1.  Run vLLM on one GPU behind its OpenAI-compatible API, and read its metrics\n    endpoint.\n\n2.  Deploy it on Kubernetes with the NVIDIA GPU Operator, and place it with\n    device selectors.\n\n3.  Put a gateway in front of it with TLS, auth and timeouts sized for long\n    streams.\n\n4.  Scrape engine and GPU metrics into Prometheus and build the core panels from\n    Section 3.8.\n\n5.  Benchmark it with a concurrency sweep, and run a small eval suite alongside\n    it.\n\n6.  Serve the same model on SGLang and compare on your own traffic.\n\n7.  Split a larger model across GPUs with tensor and pipeline parallelism, then\n    try a distributed serving layer.\n\n8.  Learn enough CUDA and profiling to understand why a kernel or a setting made\n    the difference.\n\n## 5. Learn from teams that share their work\n\nMuch of what you need to know about running inference comes from teams that\nwrote up what broke. Learn to read those write-ups critically, and write your\nown (Section 6).\n\n### How to read a case study\n\n- **Find the setup.** Model, hardware, engine version and traffic shape. Note\n  the date, because numbers age fast.\n\n- **Find the bar.** What counted as good enough, and who decided.\n\n- **Follow the failures.** What broke, how the team found out, and how long it\n  took to detect.\n\n- **Check the method.** Did they change one thing at a time and measure it on\n  their own traffic?\n\n- **Separate what transfers.** A timeout lesson applies everywhere, but a\n  specific flag value only holds for that engine release on that hardware.\n\n#### Case study: Project Champagne (2026)\n\nUpbound, the company behind Crossplane and Modelplane, self-hosted chat, coding\nagents and a CVE-fixing build agent on open models with Modelplane, across AWS\nand Nebius GPUs. The team published what broke in a five-part series.\nHighlights, with numbers as of 2026:\n\n- **Serving was the smaller problem.** vLLM and scheduling worked. TLS,\n  identity, timeouts, storage and observability took the time. (Part 2)\n- **Quota did not mean capacity.** Approved quota for 768 vCPUs of H100\n  instances never produced one, and the planned GPU arrived about half the time.\n  One model ended up split across two nodes in two zones. (Part 2)\n- **Fitting at startup wasn't enough.** A chat model that fit cleanly ran out of\n  memory nine times in four days. Lowering the memory fraction from 0.92 to\n  0.88, enabling expandable segments and capping concurrent sequences at 16\n  freed 4.6 GiB. (Part 2)\n- **The expensive failures were silent.** A truncated stream returned 200, a\n  metrics API reported available with no data, and monitoring sat pending for\n  six days. (Part 2)\n- **Timeouts stacked up.** A 15-second route default, then a 60-second load\n  balancer idle timeout, cut off answers that were still streaming. (Parts 2 and\n  3)\n- **Agents were the hardest clients.** Hour-long sessions ran past the 60-minute\n  lifetime of their tokens, and a reasoning model thought for more than a minute\n  before its first visible token. (Parts 2 and 3)\n- **Low scores were often not the model’s fault.** A fake shell in the test\n  harness scored Claude 0. A 60-second idle timeout failed a model that\n  completed every run once the limit was raised. (Part 3)\n- **Slowness had several causes.** An engineer’s “wow, is it slow” traced back\n  to hidden maximum-effort reasoning while every dashboard looked fine. One GPU\n  showed 99% `GPU-Util` with tensor cores busy about 10% of the time. (Part 4)\n- **Load depended on the client.** Agent prompts had a median of 59K tokens,\n  about two-thirds hit the prefix cache, and short-prompt sessions kept more\n  requests in flight by launching subagents. (Part 5)\n- **The engine mattered more than expected.** On the same two 8-GPU nodes,\n  moving from vLLM to SGLang was the largest single gain. A 128K context cap\n  only looked faster because it rejected long conversations. (Part 5)\n\n### Read the full series\n\nEach post on the Upbound blog tells these stories in detail, with the configs,\nqueries and numbers behind them:\n\n- [**Part 1: What it takes to own your\n  intelligence.**](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence)\n  Why Upbound self-hosts inference, and how the use cases picked the models and\n  hardware.\n\n- [**Part 2: Finding the\n  corks.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks)\n  The inference lifecycle stage by stage, from pick and place to change and\n  fail.\n\n- [**Part 3: The build security\n  agent.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent)\n  How to tell whether an open model can do a real job as well as a hosted one.\n\n- [**Part 4: Candling the\n  bottles.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles)\n  What the team could see at each step, what it missed, and how it plans to\n  alert.\n\n- [**Part 5: From pilot to\n  team.**](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team)\n  A repeatable way to test serving changes and size capacity for a growing team.\n\n\n## 6. Making the career move\n\nYou don't need to start over. Add inference concepts to the platform skills you\nhave, and show the work in public. The field is young, so newcomers can catch up\nfast.\n\n### Pick a direction\n\nPick the direction closest to the work you already do. Most platform engineers\nstart in the first two and grow toward the third.\n\n| Direction             | Focus                                                                   | Good fit if you                                  |\n|-----------------------|-------------------------------------------------------------------------|--------------------------------------------------|\n| Inference platform    | Clusters, GPU scheduling, gateways, autoscaling, multi-cloud capacity   | Enjoy Kubernetes and distributed systems         |\n| Inference reliability | SLOs, observability, capacity planning, rollouts, incident response     | Have carried a pager and run SRE practices       |\n| Runtime performance   | Engine tuning, quantization, speculative decoding, parallelism, kernels | Like profiling and working close to the hardware |\n\n### Follow a learning path\n\nVishakha Sadhwani's *AI Inference Engineering Roadmap 2026* is a thorough\n13-stage map of the field. Coming from platform engineering, you can move\nthrough it faster, because you already own much of the infrastructure half. A\nshorter path:\n\n1.  **Learn the model, not just the container.** Transformer basics, attention\n    and the KV cache, dense versus MoE models, and how to read a model config.\n    The goal is to estimate the memory any model needs in five minutes.\n\n2.  **Learn one engine deeply.** Pick vLLM, learn what each core flag trades\n    away, and measure time to first token and per-token latency yourself.\n\n3.  **Translate your platform skills to GPUs.** Device allocation, gateways that\n    hold long streams, autoscaling on inference signals, and observability for\n    engines.\n\n4.  **Learn to judge quality and cost.** Build an eval, run a capacity sweep and\n    calculate cost per million tokens. This is where platform engineers most\n    often have a gap, and where the role adds the most value.\n\n### Build projects that prove it\n\nEach project below fits on a rented GPU or two, and each produces a repo with\nreal numbers. Shut instances down between runs, and use spot capacity for batch\nexperiments.\n\n| Project                                                                                                             | What it proves                                                      | What to publish                                              |\n|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------|\n| Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users.                                        | You can measure TTFT, time per output token and throughput accurately | Configs, the load script, charts and cost per million tokens |\n| Predict how many 32K-token sessions fit on your GPU, then test it                                                   | You understand weights, KV cache and runtime overhead               | Your prediction, the result and why they differ              |\n| Compare FP16 and FP8, and a 4-bit format if you can                                                                 | You can trade quality against speed with evidence                   | Eval scores next to latency and throughput                   |\n| Run the same prompts with shared content first, then last                                                           | You understand prefix caching and prompt structure                  | Cache hit rate and TTFT for each                             |\n| Deploy on Kubernetes with the GPU Operator, a gateway with TLS and auth, autoscaling on queue depth, and dashboards | You can run inference as a platform                                 | Manifests or compositions, and dashboard definitions         |\n| Compare vLLM and SGLang on agent-like traffic, one change at a time                                                 | You make engine choices with data                                   | The bar, the method and results with intervals               |\n| Build an eval harness for one agent task with deterministic graders                                                 | You can tell a model failure from a harness failure                 | Tasks, graders and negative controls                         |\n| Reproduce a failure: a timeout, an out-of-memory crash or a stuck rollout                                           | You can operate it, not just deploy it                              | A short postmortem and runbook                               |\n\n### Show your work in public\n\nMany inference roles fill through networks before they reach a careers board. In\nAugust 2026, [Zeoli\nfound](https://www.datagravity.dev/p/the-rise-of-inference-engineering) one\nserving company valued at \\$17.5B that didn't list any dedicated inference\nperformance openings. Public work is how people outside those networks get\nfound.\n\n- **Write READMEs like lab reports.** State the question, the setup (model, GPU,\n  engine version and flags), the method, the results with numbers, what\n  surprised you, and how to reproduce it.\n\n- **Publish what failed.** A clear account of a failure shows more skill than a\n  clean demo.\n\n- **Pin versions and show cost.** Results without an engine version and a price\n  are hard to trust and harder to reuse.\n\n- **Write and speak about it.** Turn each project into a short post. Submit\n  talks to meetups, CNCF events and community live streams.\n\n### Contribute to open source\n\nMuch of what inference projects still need is platform work. Upbound's own\nupstream list from Project Champagne reads like a platform backlog: configurable\nload balancers, certificate issuance, Recreate rollouts for exclusive devices,\nan honest Ready signal, engine metrics scraped out of the box, and auth that\nsuits long-running agents.\n\nProjects to follow include [vLLM](https://github.com/vllm-project/vllm),\n[SGLang](https://github.com/sgl-project/sglang),\n[llm-d](https://github.com/llm-d/llm-d),\n[KServe](https://github.com/kserve/kserve), [NVIDIA\nDynamo](https://github.com/ai-dynamo/dynamo), [Agent\nRouter](https://theagentrouter.ai) (formerly Envoy AI Gateway), the [Gateway API\nInference\nExtension](https://github.com/kubernetes-sigs/gateway-api-inference-extension),\n[Modelplane](https://github.com/modelplaneai/modelplane) and\n[Crossplane](https://github.com/crossplane/crossplane). Good first\ncontributions:\n\n- **Run it and report what breaks,** with a reproduction and numbers. This is\n  how Champagne contributed most.\n\n- **Write operator documentation:** deployment, tuning and troubleshooting\n  guides drawn from your own runs.\n\n- **Add metrics, dashboards and alerts** that make silent failures loud.\n\n- **Improve manifests, Helm charts and compositions** for real environments.\n\n- **Contribute benchmark recipes and tests** for the hardware you have.\n\nRead each project's contributing guide, start with a small change, and comment\non an issue before taking on a large one.\n\n### Ongoing learning\n\nEngines add flags every few weeks, and research reaches production within\nmonths.\n\n- **Read release notes** for the engine you run, every release, and re-run your\n  benchmark when you upgrade.\n\n- **Read the core papers:**\n  [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) on continuous\n  batching, [PagedAttention](https://arxiv.org/abs/2309.06180),\n  [DistServe](https://arxiv.org/abs/2401.09670) on disaggregation, the original\n  [speculative decoding](https://arxiv.org/abs/2211.17192) paper, and\n  [FlashAttention](https://arxiv.org/abs/2205.14135).\n\n- **Read Philip Kiely's [*Inference\n  Engineering*](https://www.baseten.co/inference-engineering/).** The e-book is\n  free from Baseten.\n\n- **Follow practitioners.** Newsletters such as [The Pragmatic\n  Engineer](https://newsletter.pragmaticengineer.com),\n  [ByteByteGo](https://blog.bytebytego.com) and [Data\n  Gravity](https://www.datagravity.dev) cover the field well. Join the community\n  channels for the engines you use.\n\n- **Learn evals.** Whether an open model can replace a hosted one is decided by\n  evals, and few engineers can build good ones. Start with Anthropic's\n  [*Demystifying evals for AI\n  agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)\n  and Hamel Husain and Shreya Shankar's [evals\n  FAQ](https://hamel.dev/blog/posts/evals-faq/).\n\n- **Learn the economics.** Be able to explain cost per million tokens and cost\n  per user session to a finance partner.\n\n### Find the roles\n\nDon't expect a consistent job title yet. The same work shows up as inference\nengineer, performance engineer, model serving engineer or ML infrastructure\nengineer, and sometimes as AI platform engineer, so search for all of them. Pay\nreflects how scarce the skills are: in August 2026, [Zeoli\nfound](https://www.datagravity.dev/p/the-rise-of-inference-engineering) an\nAnthropic inference performance role posted at \\$350,000 to \\$850,000, and an\nOpenAI equivalent at \\$295,000 to \\$555,000.\n\nYou may not need to change jobs to get started, though. If your company is\nplanning to self-host its first model, put your hand up. Your team will probably\nend up responsible for that project anyway, and running it is the most direct\nexperience you can get.\n\n### Prepare for interviews\n\nPractice answering these. Each one points to the section of this guide that\ncovers it. Several are adapted from Lamhot Siagian's interview pocket notes on\nefficient inference (Section 7).\n\n- **Prefill and decode.** Why is decode usually limited by memory bandwidth\n  rather than compute, and what changes at long context? (Section 2)\n\n- **Hardware trends.** If compute keeps growing faster than memory bandwidth,\n  will inference soon become compute-bound? (Section 2)\n\n- **Memory math.** How much memory do a model and its KV cache need at a given\n  context length and concurrency? (Section 2)\n\n- **Goodput.** What is goodput, and why plan capacity with it? (Section 2)\n\n- **Prefix caching.** Which applications benefit most from prefix caching?\n  (Section 2)\n\n- **Precision.** You need to serve a 70B model on two 80 GB GPUs. What precision\n  would you choose, and why? (Sections 2 and 3.4)\n\n- **Quantization.** When does weight-only quantization speed things up, and when\n  does it not? (Section 3.4)\n\n- **Speculative decoding.** When would you not use speculative decoding?\n  (Section 3.4)\n\n- **Chunked prefill.** How do long prompts slow other users' streams, and what\n  does chunked prefill change? (Section 3.4)\n\n- **Parallelism.** Which parallelism strategy would you pick for a given model\n  and GPU type, and when does tensor parallelism beat pipeline parallelism?\n  (Sections 3.4 and 3.5)\n\n- **Profiling.** How would you tell whether a workload is limited by compute or\n  by memory bandwidth? (Section 3.8)\n\n- **Debugging \"slow.\"** p99 time to first token just spiked. How do you tell\n  queueing, hidden reasoning and slow generation apart? (Section 3.8)\n\n- **Cost.** What does a million tokens cost on your GPUs, and how does that\n  compare with a hosted API? (Section 2)\n\n- **Capacity planning.** Your user count will grow 10x in six months. How do you\n  plan capacity? (Section 3.10)\n\n- **Rollouts and SLOs.** How would you roll out a new model version safely, and\n  what SLO would you set for the service? (Sections 3.8 and 3.10)\n\n- **Failure stories.** Describe one failure with real numbers. Cover what broke\n  and how you found it, then what you changed.\n\n## 7. Sources and further reading\n\n\n### Upbound and Modelplane\n\n- Upbound, *Drinking Our Own Champagne* series on the Upbound blog: [Part 1:\n  What it takes to own your\n  intelligence](https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence);\n  [Part 2: Finding the\n  corks](https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks);\n  [Part 3: Use case \\#1, the build security\n  agent](https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent);\n  [Part 4: Candling the\n  bottles](https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles);\n  [Part 5: What does it take to grow a self-hosted coding model from pilot to\n  team?](https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team)\n\n- Viktor Farcic, *LLM Inference Explained: 12 Concepts You Actually Need to\n  Know*.\n\n- [Modelplane on GitHub](https://github.com/modelplaneai/modelplane) and\n  [Modelplane documentation](https://docs.modelplane.ai).\n\n### Books and articles\n\n- Philip Kiely, [*Inference\n  Engineering*](https://www.baseten.co/inference-engineering/), Baseten Books,\n  2026. Free as an e-book.\n\n- Gergely Orosz & Philip Kiely, [*What is inference engineering?\n  Deepdive*](https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering),\n  The Pragmatic Engineer, March 31, 2026.\n\n- ByteByteGo, [*A Guide to AI Inference\n  Engineering*](https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering),\n  June 15, 2026.\n\n- Chris Zeoli, [*The Rise of Inference\n  Engineering*](https://www.datagravity.dev/p/the-rise-of-inference-engineering),\n  Data Gravity, August 13, 2026.\n\n- Spheron, [*What Is Inference Engineering? The 2026 GPU Cloud\n  Guide*](https://www.spheron.network/blog/inference-engineering-guide-2026/),\n  updated October 19, 2026.\n\n- Vishakha Sadhwani, [*AI Inference Engineering Roadmap 2026: Complete Learning\n  Path*.](https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap)\n\n- Charity Majors, Liz Fong-Jones and George Miranda, [*Observability\n  Engineering*](https://www.oreilly.com/library/view/observability-engineering/9781492076438/),\n  O'Reilly.\n\n- Lamhot Siagian, Efficient LLM Inference: Interview Pocket Notes, second\n  edition, 2026.\n\n### Evals and benchmarks\n\n- Anthropic, [*Demystifying evals for AI\n  agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents),\n  January 2026.\n\n- Hamel Husain and Shreya Shankar, [*AI Evals\n  FAQ*](https://hamel.dev/blog/posts/evals-faq/).\n\n- UK AI Security Institute, [Inspect](https://inspect.aisi.org.uk).\n\n- Hugging Face, [upskill](https://github.com/huggingface/upskill).\n\n- SemiAnalysis, [AgentX methodology and\n  datasets](https://inferencex.semianalysis.com/agentx/methodology).\n\n### Papers\n\n- Yu et al., [Orca: A Distributed Serving System for Transformer-Based\n  Generative Models](https://www.usenix.org/conference/osdi22/presentation/yu),\n  OSDI 2022.\n\n- Kwon et al., [Efficient Memory Management for Large Language Model Serving\n  with PagedAttention](https://arxiv.org/abs/2309.06180), SOSP 2023.\n\n- Zhong et al., [DistServe: Disaggregating Prefill and Decoding for\n  Goodput-optimized Large Language Model\n  Serving](https://arxiv.org/abs/2401.09670), OSDI 2024.\n\n- Leviathan et al., [Fast Inference from Transformers via Speculative\n  Decoding](https://arxiv.org/abs/2211.17192), ICML 2023.\n\n- Dao et al., [FlashAttention](https://arxiv.org/abs/2205.14135), 2022, and Shah\n  et al., [FlashAttention-3](https://arxiv.org/abs/2407.08608), 2024.\n\n- Shoeybi et al., [Megatron-LM: Training Multi-Billion Parameter Language Models\n  Using Model Parallelism](https://arxiv.org/abs/1909.08053), 2019.\n\n- Shazeer et al., [Outrageously Large Neural Networks: The Sparsely-Gated\n  Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538), 2017.\n\n","body_html":"<h1 id=\"an-open-field-guide-to-inference-engineering\">An Open Field Guide to Inference Engineering</h1>\n<p>A field guide to inference concepts, production operations, tooling, and career development for platform engineers.</p>\n<p>Source: <a href=\"https://docs.modelplane.ai/guides/inference-guide/\" rel=\"nofollow ugc noopener\">https://docs.modelplane.ai/guides/inference-guide/</a></p>\n<p><em>Numbers, prices and tool versions are as of October 2026.</em></p>\n<p>This guide is for platform engineers new to inference. If you need to understand\nhow models and GPUs work together and how to run inference in production, this\nguide will help. You&#39;ll also learn about the tools to use and how to grow into\nan inference engineering role.</p>\n<p>This guide builds on the work of others in the inference community. If you&#39;re\ninterested in inference engineering, continue your learning with these:</p>\n<ul><li><p>Viktor Farcic&#39;s [*LLM Inference</p><p>Explained*](<a href=\"https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/\" rel=\"nofollow ugc noopener\">https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/</a>)</p></li><li><p>Philip Kiely&#39;s [*Inference</p><p>Engineering*](<a href=\"https://www.baseten.co/inference-engineering/\" rel=\"nofollow ugc noopener\">https://www.baseten.co/inference-engineering/</a>)</p></li><li><p>[The Pragmatic Engineer, What Is Inference</p><p>Engineering?](<a href=\"https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering\" rel=\"nofollow ugc noopener\">https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering</a>)</p></li><li><p>[ByteByteGo&#39;s Guide to AI Inference</p><p>Engineering](<a href=\"https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering\" rel=\"nofollow ugc noopener\">https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering</a>)</p></li><li><p>[Data Gravity on The Rise of Inference</p><p>Engineering](<a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">https://www.datagravity.dev/p/the-rise-of-inference-engineering</a>)</p></li><li><p>[Spheron&#39;s Inference Engineering</p><p>Guide](<a href=\"https://www.spheron.network/blog/inference-engineering-guide-2026/\" rel=\"nofollow ugc noopener\">https://www.spheron.network/blog/inference-engineering-guide-2026/</a>)</p></li><li><p>Vishakha Sadhwani&#39;s [*The AI Inference Engineering</p><p>Roadmap*](<a href=\"https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap\" rel=\"nofollow ugc noopener\">https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap</a>)</p></li><li><p>Upbound, [*Project</p><p>Champagne*](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence</a>)\nseries</p></li></ul>\n<h2 id=\"1-why-now\">1. Why now?</h2>\n<p>In an AI-forward organization, platform teams become inference teams. Coding\nassistants can&#39;t send source code to third parties, and frontier model bills\nincrease month to month. Support tools also need internal data that a hosted\nmodel can&#39;t reach.</p>\n<p>Organizations want to run these models themselves, and open models now make that\npractical.</p>\n<p><a href=\"https://huggingface.co/models\" rel=\"nofollow ugc noopener\">Hugging Face</a> hosts more than two million open\nmodels, and the best of them compete with closed models. DeepSeek released R1 in\nJanuary 2025, and <a href=\"https://arxiv.org/abs/2501.12948\" rel=\"nofollow ugc noopener\">it matched OpenAI&#39;s o1 on math and coding benchmarks about a\nmonth after o1 came out</a>. When you run your\nown model, you control latency, uptime, and where your data goes. At high\nvolume, you can also <a href=\"https://www.baseten.co/inference-engineering/\" rel=\"nofollow ugc noopener\">cut costs by about\n80%</a>.</p>\n<p>The work suits platform engineers. The price of a fixed level of AI capability\nfalls roughly 10x a year, and most of that comes from how models are served,\n<a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">not from better\nGPUs</a>. Upbound,\nthe company behind Crossplane and Modelplane, saw this firsthand. In its\n<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence\" rel=\"nofollow ugc noopener\">Project\nChampagne</a>\nblog series, the team ran its own models on\n<a href=\"https://github.com/modelplaneai/modelplane\" rel=\"nofollow ugc noopener\">Modelplane</a> and found that vLLM and\nscheduling worked. The time went to TLS, identity, timeouts, storage, IAM and\nobservability.</p>\n<h3 id=\"skills-that-carry-over\">Skills that carry over</h3>\n<p>An inference engineer picks a model, gets it onto the right GPU, keeps it\nserving, makes it faster and cheaper, and gets paged when it breaks. That&#39;s SRE\nfor a new workload, and most of your skills carry over:</p>\n<div class=\"table-wrap\"><table><thead><tr><th>What you already know</th><th>What it becomes in inference</th></tr></thead><tbody><tr><td>Scheduling and node pools</td><td>GPU placement with device selectors, taints and exclusive devices</td></tr><tr><td>Capacity planning</td><td>GPU memory math: weights, KV cache and runtime overhead, checked at runtime</td></tr><tr><td>Load balancing</td><td>Cache-aware routing and long-lived streaming connections</td></tr><tr><td>Autoscaling on CPU</td><td>Scaling on queue depth and KV cache use, with cold starts measured in minutes</td></tr><tr><td>Rollouts</td><td>Rollouts of exclusive GPUs, and a version matrix of model, hardware and engine build</td></tr><tr><td>SLOs and latency percentiles</td><td>Time to first token, time per output token and goodput</td></tr><tr><td>Observability</td><td>Engine metrics, real GPU activity and speculative decoding acceptance</td></tr><tr><td>FinOps</td><td>Cost per million tokens and cost per user session</td></tr><tr><td>Identity and access</td><td>Credentials that last as long as an agent session</td></tr></tbody></table></div>\n<p>A hosted API is still <a href=\"https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering\" rel=\"nofollow ugc noopener\">the right call early in a product&#39;s\nlife</a>.\nSelf-hosting pays off when API spend becomes a real line item, when latency or\nreliability needs outgrow vendors, or when data can&#39;t leave your network. In\nAugust 2026, <a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">Chris Zeoli\ncounted</a> about\n58 open inference roles on public careers boards, with no settled job title.</p>\n<h2 id=\"2-inference-101\">2. Inference 101</h2>\n<p>This section covers the concepts in roughly the order you&#39;ll meet them. Most are\nabout what the engine keeps in GPU memory or what limits its speed.</p>\n<p><a href=\"https://devopstoolkit.live/ai/llm-inference-explained-12-concepts-you-actually-need-to-know/\" rel=\"nofollow ugc noopener\">Viktor\nFarcic</a>\nsuggests asking when each concept starts to matter. Some matter as soon as you\ndeploy, others only once many people use the model, and some you may never need.\nThe table near the end of this section gives the signal for each.</p>\n<h3 id=\"what-an-inference-engine-is\">What an inference engine is</h3>\n<p>An inference engine, like <a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM</a> or\n<a href=\"https://github.com/sgl-project/sglang\" rel=\"nofollow ugc noopener\">SGLang</a>, is the server that turns model\nweights into an API. For a platform engineer, the closest mental model is a\nstateful database server with a GPU attached. It loads a large dataset into\nmemory at startup and keeps per-session state. It also schedules work internally\nand answers over HTTP. <a href=\"https://github.com/ollama/ollama\" rel=\"nofollow ugc noopener\">Ollama</a> on a laptop is\nthe same idea at small scale, and every hosted model API runs one behind the\nscenes.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>You know this from databases and services</th><th>The engine&#39;s counterpart</th></tr></thead><tbody><tr><td>Loading a dataset into the buffer pool at startup</td><td>Loading model weights into GPU memory, which can take many minutes</td></tr><tr><td>Per-session state in a session store</td><td>The KV cache, one for each conversation in flight</td></tr><tr><td>A query planner and connection pool</td><td>The scheduler, which batches requests into each GPU pass</td></tr><tr><td>A wire protocol clients already speak</td><td>An OpenAI- or Anthropic-compatible HTTP API</td></tr><tr><td>Warming caches after a restart</td><td>Cold starts: pulling a large image and reloading the weights</td></tr></tbody></table></div>\n<p>Unlike a database, the engine keeps its working set in GPU memory, not RAM. GPU\nmemory is small and expensive, so most tuning is about deciding how much of it\neach request gets.</p>\n<h3 id=\"the-inference-stack\">The inference stack</h3>\n<p>The engine is one layer of four. At the bottom, <strong>kernels</strong> do the arithmetic.\nYou rarely choose them directly. Above them sits the <strong>engine</strong>. Above that, a\n<strong>control plane</strong> deploys and manages engines, because sooner or later you run\nmore than one. On top, a <strong>gateway</strong> routes traffic across models and replicas.</p>\n<p><em>Inference stack: four layers and the clients they serve</em></p>\n<p>Requests pass through the gateway to an engine, which runs kernels on the GPU.\nThe control plane deploys and manages the gateway and engines, but no request\npasses through it.</p>\n<h3 id=\"tokens\">Tokens</h3>\n<p>A token is the unit a language model works in, roughly a word or word fragment.\nContext windows, prices and speeds are all counted in tokens. When someone\nquotes tokens per second, they mean this unit.</p>\n<h3 id=\"weights-and-the-kv-cache\">Weights and the KV cache</h3>\n<p>The weights are fixed, and the KV cache grows with load. Size for both, as you\nwould a database&#39;s footprint and working set.</p>\n<p><strong>Weights are the fixed footprint.</strong> Most models are released at 2 bytes per\nparameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live\nin GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of\nthe card for everything else. That number is the same for one user or a hundred.</p>\n<p><strong>The KV cache is the working set.</strong> Each new token is computed against every\ntoken before it. Rather than recompute that history, the engine keeps the\nattention keys and values for every token in flight. Its size scales with\ncontext length multiplied by the number of concurrent conversations, much like a\nsession store that grows with both session size and user count.</p>\n<p>You can estimate the KV cache per token from a model&#39;s config:</p>\n<pre><code class=\"language-text\">KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value</code></pre>\n<p>For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is\nabout 320 KB per token. One 128K-token conversation needs about 40 GB of cache.\nAt that size, the KV cache usually limits how many users a GPU can serve.</p>\n<p>Watch for these sizing mistakes:</p>\n<ul><li><p><strong>Mixture-of-experts (MoE) models keep every weight resident.</strong> A model with</p><p>320B total parameters and 18B active still needs about 320 GB at FP8. Only\ncompute scales with the active parameters.</p></li><li><p><strong>Read the checkpoint size, not the dtype table.</strong> While self-hosting its own</p><p>models for <a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks\" rel=\"nofollow ugc noopener\">Project\nChampagne</a>,\nUpbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The\nreal checkpoint was 177.7 GB, because the experts were stored in a 4-bit\nformat.</p></li><li><p><strong>Leave room for runtime overhead.</strong> CUDA graphs, activations and memory</p><p>fragmentation all take space. A model that fits at startup can still run out\nof memory days later (<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks\" rel=\"nofollow ugc noopener\">Project Champagne, Part\n2</a>).</p></li></ul>\n<h3 id=\"two-phases-two-bottlenecks\">Two phases, two bottlenecks</h3>\n<p>Each request runs in two phases on the same GPU, and <a href=\"https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering\" rel=\"nofollow ugc noopener\">they are limited by\ndifferent\nthings</a>.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Phase</th><th>What it does</th><th>Limited by</th><th>Metric it sets</th></tr></thead><tbody><tr><td>Prefill</td><td>Reads the whole input in parallel, including the system prompt and tool definitions as well as the conversation history. Produces the first token and the KV cache.</td><td>Compute</td><td>Time to first token (TTFT)</td></tr><tr><td>Decode</td><td>Writes the answer one token at a time. Each token needs a pass that reads all active weights, plus the KV cache, from memory. At long context, reading the cache can cost more than reading the weights.</td><td>Memory bandwidth</td><td>Time per output token and tokens per second</td></tr></tbody></table></div>\n<p>Generating one token does one or two floating-point operations per byte read\nfrom memory, depending on weight precision. An H100 needs about 295 operations\nper byte to stay busy. That means one request uses a small fraction of the chip\nwhile the compute units wait on memory (<a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">Data\nGravity</a>).</p>\n<p>Peak compute has grown about 3x every two years, against about 1.6x for memory\nbandwidth. Nearly every optimization either moves fewer bytes per token or\nspreads each byte across more tokens.</p>\n<p>Because the phases have opposite bottlenecks, a technique that speeds one often\ndoes little for the other. That is why benchmarks report TTFT and tokens per\nsecond separately.</p>\n<h3 id=\"metrics-to-track\">Metrics to track</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>Metric</th><th>What it measures</th><th>Watch out for</th></tr></thead><tbody><tr><td>Time to first token (TTFT)</td><td>Queueing plus prefill: how long the user waits before anything appears</td><td>With reasoning models, the engine&#39;s first token may be hidden thinking. Measure the first token the user sees.</td></tr><tr><td>Time per output token, or inter-token latency</td><td>Decode speed per stream</td><td>Speculative decoding and memory bandwidth show up here</td></tr><tr><td>Tokens per second per user</td><td>Interactivity, or the model&#39;s typing speed</td><td>Falls as batch size rises</td></tr><tr><td>Throughput</td><td>Total tokens per second across all users</td><td>Rises as batch size rises</td></tr><tr><td>Goodput</td><td>Throughput that still meets your latency targets</td><td>The number to plan capacity with</td></tr><tr><td>End-to-end latency</td><td>Total request time</td><td>Depends on how much the model writes. A healthy 190-second answer is normal for long generations.</td></tr><tr><td>Percentiles</td><td>p50, p90, p99 of each latency</td><td>Users feel the tail, not the average</td></tr><tr><td>Errors and refusals</td><td>Failed or rejected requests</td><td>Count refusals as errors, or a context cap can look like a speedup</td></tr><tr><td>Cost per million tokens</td><td>GPU cost divided by tokens served</td><td>Compare at the load you actually run</td></tr></tbody></table></div>\n<p>Cost per million tokens converts a GPU price and a throughput into one\ncomparable number:</p>\n<pre><code class=\"language-text\">Cost per million tokens = GPU cost per hour / (tokens per second × 3600) × 1,000,000</code></pre>\n<p><a href=\"https://www.spheron.network/blog/inference-engineering-guide-2026/\" rel=\"nofollow ugc noopener\">Spheron&#39;s</a>\nworked example: a GPU at \\$2.90 an hour sustaining 1,200 tokens per second costs\nabout \\$0.67 per million tokens.</p>\n<h3 id=\"latency-versus-throughput\">Latency versus throughput</h3>\n<p>Batching more requests together raises total throughput and slows each\nindividual user. Interactive chat and coding favor latency. Overnight batch jobs\nfavor throughput.</p>\n<p>It&#39;s the same trade-off as tuning a database for transactions or for analytics.\nWrite down the workload&#39;s constraints before you tune anything.</p>\n<h3 id=\"when-to-adopt-each-concept\">When to adopt each concept</h3>\n<p>Each row gives the symptom that tells you it&#39;s time to act, in roughly the order\nteams need them.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Concept</th><th>What it does</th><th>The signal that you need it</th></tr></thead><tbody><tr><td>Fitting the weights</td><td>Choose a model, precision and GPU so the weights fit with room to spare</td><td>The engine fails to start, or starts with almost no KV cache left</td></tr><tr><td>Continuous batching and paged KV cache</td><td>Share each GPU pass across requests, and hand out cache in small blocks</td><td>Already on by default in vLLM and SGLang. Check it, don&#39;t build it.</td></tr><tr><td>Quantization</td><td>Store weights in fewer bits, and give the freed memory to the KV cache</td><td>Requests queue while the GPU still has compute to spare</td></tr><tr><td>Prefix caching</td><td>Reuse the work already done on repeated prompt openings</td><td>Already on by default in vLLM and SGLang. Watch the hit rate once your clients are agents or chat apps that resend long histories</td></tr><tr><td>Speculative decoding</td><td>Draft several tokens and verify them in one pass</td><td>Individual responses stream too slowly, even at low load</td></tr><tr><td>Gateway and control plane</td><td>Route between replicas, and manage engines declaratively</td><td>You add a second model, a second replica or a second cluster</td></tr><tr><td>Sharding across GPUs</td><td>Split one model with tensor, pipeline or expert parallelism</td><td>The model will not fit on one GPU. Plan this before you deploy.</td></tr><tr><td>Autoscaling</td><td>Add and remove replicas on queue depth and KV cache use</td><td>Idle GPUs show up on the bill, or queues build at peak hours</td></tr><tr><td>Disaggregation</td><td>Run prefill and decode on separate GPUs</td><td>Long prompts stall streaming at scale, and you have RDMA-class networking</td></tr></tbody></table></div>\n<p>Most teams need only the first few rows on day one. Add the rest when their\nsignals show up.</p>\n<h3 id=\"what-agents-change\">What agents change</h3>\n<p>Coding agents and other agentic workloads stress every part of this picture.\nUpbound measured this in <a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team\" rel=\"nofollow ugc noopener\">Part\n5</a>\nof its Project Champagne series (introduced in Section 1), where about 10 of its\nengineers ran Claude Code and other coding agents against a self-hosted model.\nTheir traffic looked like this:</p>\n<ul><li><strong>Prompts are long.</strong> The median prompt was 59K tokens and the p90 was 125K.</li><li><p><strong>Prompts are mostly repeated.</strong> About two-thirds of prompt tokens hit the</p><p>prefix cache, because agents resend their transcript on every turn.</p></li><li><p><strong>Thinking is hidden.</strong> A reasoning model can think for more than a minute</p><p>before its first visible token. Every timeout between the agent and the model\nhas to allow for that.</p></li><li><p><strong>Load depends on the client.</strong> Short-prompt sessions launch subagents and can</p><p>keep more requests in flight than long-prompt sessions. Plan capacity in\nrequests in flight rather than seats.</p></li><li><p><strong>Sessions run for hours.</strong> Streams, tool calls and credentials all have to</p><p>keep working for that long.</p></li></ul>\n<h2 id=\"3-running-inference-in-production\">3. Running inference in production</h2>\n<p>In production, every choice trades latency, throughput, cost and quality against\neach other, usually with too few GPUs. Where a practice comes from Project\nChampagne, Section 5 has the details.</p>\n<h3 id=\"3-1-start-from-use-cases-not-hardware\">3.1 Start from use cases, not hardware</h3>\n<p>Each use case sets different requirements. In Project Champagne:</p>\n<ul><li><p><strong>Coding agents</strong> needed long context without quality loss, reliable tool-call</p><p>parsing, streaming, sessions that last hours, and an Anthropic-compatible API\nfor Claude Code.</p></li><li><p><strong>Chat</strong> needed low time to first token, real concurrency, and an</p><p>OpenAI-compatible API for existing tools.</p></li><li><p><strong>The platform</strong> needed TLS on a real hostname and identity tied to the</p><p>company domain. It also needed observability that tells a healthy engine from\na busy one, and correct placement without babysitting.</p></li></ul>\n<p>Separate interactive workloads from batch workloads. They need different\ntargets, and often different deployments. Write the pass and fail bar down\nbefore you test anything.</p>\n<h3 id=\"3-2-pick-models-deliberately\">3.2 Pick models deliberately</h3>\n<ul><li><p><strong>Evaluate on your own tasks.</strong> Public benchmarks measure someone else&#39;s</p><p>workload. Section 3.11 covers how.</p></li><li><p><strong>Start small, then grow.</strong> A small MoE model with a few billion active</p><p>parameters builds operating experience faster and costs less to get wrong.</p></li><li><p><strong>The newest model is not always the right one.</strong> Model releases move faster</p><p>than hardware plans. A new ‘small’ flagship can need several of the largest\ncards, while your current model may be running at half its supported context.\nThe cheapest gain may not need a new model at all.</p></li><li><p><strong>Check that your engine can actually load it.</strong> A model listed as supported</p><p>in release notes may still fail to load on your hardware and engine build.</p></li><li><p><strong>Check the license.</strong> Most open models use permissive licenses, but some</p><p>restrict commercial use.</p></li><li><p><strong>Check the chat template and parsers.</strong> Tool calls and reasoning output</p><p>depend on settings each model needs in the engine. A wrong one breaks tool use\nwhile every request still returns 200.</p></li></ul>\n<h3 id=\"3-3-size-for-runtime-not-startup\">3.3 Size for runtime, not startup</h3>\n<p>A model that fits at startup can still run out of memory under real traffic when\nfragmentation pushes past the profiled peak. That can happen days later, in the\nmiddle of a request.</p>\n<ul><li><p><strong>Budget all three costs:</strong> weights, KV cache, and runtime overhead such as</p><p>CUDA graphs and activations.</p></li><li><p>**Know what your engine&#39;s memory setting covers, and check again on every</p><p>upgrade.** vLLM sizes its KV cache from a startup estimate of everything else,\nincluding CUDA graph pools, and real traffic can go past that estimate. If you\nsee out-of-memory errors under load, lower the memory fraction slightly,\nenable expandable segments and cap concurrent sequences, then measure free\nmemory again.</p></li><li><p><strong>Cap every request.</strong> Set a maximum output length on every call. Without one,</p><p>any request can take a model down for everyone, and abandoned requests keep\ngenerating after the client gives up.</p></li><li><p><strong>Shorten context with care.</strong> A shorter maximum context frees KV cache for</p><p>more users. It also rejects every longer conversation, so count those\nrejections as errors.</p></li></ul>\n<h3 id=\"3-4-know-the-optimization-techniques\">3.4 Know the optimization techniques</h3>\n<p>Most engines turn on continuous batching and paged KV cache by default. The rest\nare choices you make per model and workload.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Technique</th><th>What it improves</th><th>Cost or risk</th><th>Use it when</th></tr></thead><tbody><tr><td>Continuous batching</td><td>Throughput. <a href=\"https://www.usenix.org/conference/osdi22/presentation/yu\" rel=\"nofollow ugc noopener\">Orca</a> reported 36.9x over a static baseline at the same latency.</td><td>Per-user speed drops as batches grow</td><td>Always. It is the default in modern engines.</td></tr><tr><td>Paged KV cache (<a href=\"https://arxiv.org/abs/2309.06180\" rel=\"nofollow ugc noopener\">PagedAttention</a>)</td><td>Concurrency. vLLM cut KV cache waste from 60 to 80% to under 4%.</td><td>Little in practice</td><td>Always. It is the default in vLLM.</td></tr><tr><td>Prefix caching</td><td>TTFT and prefill cost on repeated openings</td><td>Only helps up to the first token that differs</td><td>System prompts, RAG, multi-turn chat and agents</td></tr><tr><td>Weight quantization</td><td>Decode always, and prefill only when activations are quantized too, as with FP8. Weight-only formats such as AWQ and GPTQ speed up decode alone. One step down in precision is typically 30 to 50% faster (Inference Engineering).</td><td>Quality loss, which you must measure</td><td>Use the lowest precision your hardware runs natively and your evals accept. FP8 runs natively on Ada, Hopper and Blackwell GPUs.</td></tr><tr><td>KV cache quantization</td><td>More cache fits, and it moves faster</td><td>Errors compound from token to token</td><td>When KV cache is the limit</td></tr><tr><td><a href=\"https://arxiv.org/abs/2211.17192\" rel=\"nofollow ugc noopener\">Speculative decoding</a>, multi-token prediction</td><td>Tokens per second at low batch sizes</td><td>Wasted compute when guesses are rejected. Its value shrinks as load rises and spare compute runs out.</td><td>Latency matters and output is predictable</td></tr><tr><td><a href=\"https://arxiv.org/abs/1909.08053\" rel=\"nofollow ugc noopener\">Tensor parallelism</a></td><td>Fits large models, faster per user</td><td>Needs fast GPU-to-GPU links such as NVLink</td><td>Large models within one node</td></tr><tr><td>Expert parallelism</td><td>Throughput for <a href=\"https://arxiv.org/abs/1701.06538\" rel=\"nofollow ugc noopener\">MoE</a> models</td><td>Routing between experts</td><td>MoE models, including across nodes</td></tr><tr><td>Pipeline parallelism</td><td>Fits a model across nodes on ordinary networking</td><td>Adds latency to each pass</td><td>The model does not fit in one node</td></tr><tr><td>Prefill and decode disaggregation</td><td>Removes interference between phases. <a href=\"https://arxiv.org/abs/2401.09670\" rel=\"nofollow ugc noopener\">DistServe</a> reported 7.4x more requests within latency limits.</td><td>Needs RDMA-class networking to move the KV cache</td><td>Large scale, long prompts, fast interconnect</td></tr></tbody></table></div>\n<p>A few rules of thumb:</p>\n<ul><li><p><strong>Structure prompts for the cache.</strong> Prefix caching stops at the first token</p><p>that differs. Put shared content such as system prompts and tool definitions\nfirst, and variable user input last. <a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">Hosted APIs price cached input at 10 to\n50% of fresh input, as of 2026, for the same\nreason</a>).</p></li><li><p><strong>Quantize from least to most sensitive.</strong> Linear weights tolerate</p><p>quantization best, then activations, then the KV cache. Attention layers are\nthe most sensitive, because small errors compound across a long sequence. Keep\nthe most sensitive parts at higher precision until your evals show you can go\nlower. Floating-point formats such as FP8 handle outlier values better than\ninteger formats.</p></li><li><p><strong>Watch speculative decoding acceptance.</strong> Acceptance depends on temperature,</p><p>subject matter and the draft model. If acceptance collapses, every pass still\npays for the guesses and the model slows while other panels look normal.</p></li><li><p><strong>Beware interactions when applying multiple techniques.</strong> Quantizing the KV</p><p>cache helps disaggregation. Larger batches leave less spare compute for\nspeculation. <a href=\"https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering\" rel=\"nofollow ugc noopener\">One Baseten engineer tried 77\nconfigurations</a>\nbefore finding one that doubled throughput for a customer (Inference\nEngineering). Change one thing at a time and measure (Section 3.10).</p></li><li><p><strong>Keep long prompts from stalling other streams.</strong> Chunked prefill splits a</p><p>long prompt across several passes, so decode keeps moving for everyone else.\nCurrent vLLM turns it on by default, and its token budget per pass is the\nsetting to tune before you consider disaggregation.</p></li></ul>\n<h3 id=\"3-5-get-gpus-and-place-them-correctly\">3.5 Get GPUs, and place them correctly</h3>\n<ul><li><p><strong>Quota doesn&#39;t guarantee capacity.</strong> Quota only lets you request instances.</p><p>Each instance size is its own capacity pool, even with the same GPU, and no\ncloud API tells you what you can actually launch.</p></li><li><p><strong>Design for whatever GPUs you can get.</strong> Expect to get the GPU you planned</p><p>for only some of the time. A model may end up split across smaller nodes in\ndifferent zones, using pipeline parallelism over ordinary networking. Allow\nmulti-zone pools, and keep the model definition independent of the instance\ntype and node count.</p></li><li><p><strong>Match models to hardware explicitly.</strong> Kubernetes Dynamic Resource</p><p>Allocation (DRA) device selectors can pin a model to a GPU architecture or a\nminimum memory size. Verify placement against what the GPU driver publishes,\nnot against a status field that echoes what you declared.</p></li><li><p><strong>Watch the cloud, not just the controller.</strong> A managed node group can report</p><p>failure while the cloud keeps trying to launch instances, and GPUs that arrive\nhours later can be shut down by a later change to the pool. Track instances\nand node joins, and list a pool&#39;s instances before you change it.</p></li><li><p><strong>Know your interconnect.</strong> SXM datacenter GPUs such as H100, H200 and B200</p><p>link GPUs with NVLink. PCIe cards such as the L40S and RTX PRO 6000 do not, so\nmulti-GPU communication goes over PCIe and host memory. That changes which\nparallelism strategies pay off.</p></li><li><p><strong>Size the system pool too.</strong> GPU nodes are tainted for inference. Monitoring,</p><p>controllers and gateways compete for whatever general capacity is left.</p></li></ul>\n<h3 id=\"3-6-gateways-routing-and-timeouts\">3.6 Gateways, routing and timeouts</h3>\n<ul><li><p><strong>Do not round-robin.</strong> One replica may already have a conversation&#39;s prefix</p><p>in cache while another has never seen it. Use cache-aware or prefix-aware\nrouting, such as the endpoint picker in the <a href=\"https://github.com/kubernetes-sigs/gateway-api-inference-extension\" rel=\"nofollow ugc noopener\">Gateway API Inference\nExtension</a>.</p></li><li><p><strong>Route by model name.</strong> One hostname can serve many models, routed on the</p><p>request&#39;s model field.</p></li><li><p><strong>Speak both common APIs.</strong> Most tools expect the OpenAI API. Claude Code</p><p>expects the Anthropic Messages API, which\n<a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM</a> serves natively, including\nstreaming and tool calls.</p></li><li><p><strong>Raise every timeout on the path.</strong> Long generations and reasoning models</p><p>break the defaults, because each hop has a default timeout sized for web\nrequests, not for generations that run for minutes. Typical culprits are a\ngateway route timeout that bounds the whole response, a load balancer idle\ntimeout, and retries that multiply the wait. Check the route, each load\nbalancer, the gateway, the client and any helper calls the client makes.</p></li><li><p><strong>Detect truncated streams.</strong> A cut stream can return HTTP 200 with valid</p><p>tokens and then stop. To the client, it looks like a finished answer.</p></li><li><p><strong>Meter usage per caller.</strong> Token-based rate limits and quotas at the gateway</p><p>keep one heavy user or runaway agent from starving everyone else.</p></li></ul>\n<h3 id=\"3-7-security-and-identity\">3.7 Security and identity</h3>\n<ul><li><p><strong>Never expose an open endpoint.</strong> Apply your auth policy before the gateway</p><p>gets an address, and remove it last when you tear down.</p></li><li><p><strong>Use real TLS on a real hostname,</strong> with a plan for certificate issuance and</p><p>renewal.</p></li><li><p><strong>Plan for three kinds of client.</strong> People are the simplest case. Browsers</p><p>cannot attach a bearer token to page loads or preflight requests, so handle\nbrowser login with OIDC at the gateway. Agents run for hours and read their\ncredential once, so a 60-minute identity token will end them mid-session. Give\nagents long-lived API keys or a credential that refreshes.</p></li><li><p><strong>Do not stack two expiring sessions.</strong> An outer gateway login in front of an</p><p>app with its own login causes an outage when the outer session expires.</p></li><li><p><strong>Test what should be refused.</strong> Build a probe matrix of requests the gateway</p><p>must reject, such as a missing, garbage, unsigned, expired or revoked key, or\na forged caller header. Include paths that must remain reachable, such as the\nhealth check. Run it on a schedule and in a loop during changes.</p></li><li><p><strong>Carry identity into usage records,</strong> so you can see who sends what.</p><p>Overwrite any caller header a client tries to forge.</p></li><li><p>Treat model weights like code. Prefer safetensors over pickle-based</p><p>checkpoints, don&#39;t enable remote code from a model repo you haven&#39;t read, and\npin each model to a specific revision.</p></li></ul>\n<h3 id=\"3-8-observability\">3.8 Observability</h3>\n<ul><li><p><strong>Scrape the engines.</strong> Engines expose detailed Prometheus metrics. Nothing</p><p>collects them unless something is set up to scrape them, such as a\nServiceMonitor and a named port.</p></li><li><p><strong>Build the core panels:</strong> requests running against the concurrency limit,</p><p>requests waiting, KV cache use, TTFT, time per output token, output tokens per\nrequest, prefix cache hit rate, speculative decoding acceptance, errors by\nstatus code, and cancellations.</p></li><li><p>Measure quality on the server. Broken tool calls, cut-off answers and empty</p><p>responses usually reach you as user complaints while the serving dashboards\nlook normal. Count them on the server instead, including tool calls that fail\nto parse, responses cut off by the length limit, empty outputs, refusals and\nhidden reasoning tokens per request. The gateway sees every response, so it\ncan count anything the engine doesn&#39;t export.</p></li><li><p><strong>Look past <code>GPU-Util</code>.</strong> It reports whether anything ran, not how much of the</p><p>chip worked. A GPU can report near 100% <code>GPU-Util</code> while SM activity is around\nhalf and tensor cores sit mostly idle. Graph SM activity and tensor activity\nas well.</p></li><li><p><strong>Tell the kinds of slow apart.</strong> When requests are queued, running is at the</p><p>limit, waiting is above zero and the KV cache is full. When the model is\nthinking, TTFT looks fine but each request writes many hidden tokens. When\ngeneration is slow, time per output token is up.</p></li><li><p><strong>Watch the watcher.</strong> Something outside the monitoring stack must check that</p><p>data is fresh and that new metrics arrive from every cluster. A metrics\nadapter can report its API as available while returning empty results.</p></li><li><p><strong>Keep logs you can debug from.</strong> Metric scrapes can make up nearly all of a</p><p>retained engine log. Keep per-request access logs with the caller, send them\nto a log store, and remember <code>kubectl logs --previous</code> after a crash.</p></li><li><p><strong>Normalize metric names across engines,</strong> so a vLLM deployment and an SGLang</p><p>deployment are judged on the same signals.</p></li></ul>\n<p>Set SLOs on what users feel, measured per request at the front door:</p>\n<ul><li><p><strong>Responsive:</strong> time to the first token the user actually sees, under a</p><p>threshold.</p></li><li><strong>Fast enough:</strong> time per output token, under a threshold.</li><li><p><strong>Answered:</strong> the request completed without an error, and any tool call in it</p><p>parsed.</p></li></ul>\n<p>Do not set an SLO on total request time, because it depends on how much the\nmodel writes. Low-traffic models need a fleet-wide SLO, longer windows or\nsynthetic traffic. Missing data reads as 100% good, so keep a separate alert for\nmetrics that stop arriving. Open source tools such as\n<a href=\"https://github.com/slok/sloth\" rel=\"nofollow ugc noopener\">Sloth</a> and\n<a href=\"https://github.com/pyrra-dev/pyrra\" rel=\"nofollow ugc noopener\">Pyrra</a> turn SLO definitions into burn-rate\nalerts. For background, see <a href=\"https://www.oreilly.com/library/view/observability-engineering/9781492076438/\" rel=\"nofollow ugc noopener\">*Observability\nEngineering*</a>.</p>\n<h3 id=\"3-9-autoscaling-and-cold-starts\">3.9 Autoscaling and cold starts</h3>\n<ul><li><p><strong>Scale on inference signals,</strong> such as queue depth, KV cache use and requests</p><p>in flight. CPU usage tells you almost nothing here.</p></li><li><p><strong>Combine traffic and load.</strong> Traffic lets you scale ahead of demand, and load</p><p>lags behind it. The settings are the ones you already tune on any autoscaler.\nSet minimum and maximum replicas and the length of the metrics window. Then\nset the scale-down delay and how many concurrent requests one replica should\nhold. <a href=\"https://www.baseten.co/inference-engineering/\" rel=\"nofollow ugc noopener\">Kiely&#39;s book</a> covers\neach in depth for inference.</p></li><li><p><strong>Plan for slow cold starts.</strong> A new GPU node must boot, pull a large engine</p><p>image and load the weights. Loading weights alone can take half an hour for a\nlarge model. Cache weights on shared storage in each cluster so new replicas\ndo not download them again.</p></li><li><p><strong>Be careful with scale-to-zero.</strong> By the time a node is ready, the spike may</p><p>be over, and scarce GPU capacity may not come back.</p></li><li><p><strong>Plan for multiple clusters at scale.</strong> Past a few hundred GPUs, capacity</p><p>pushes workloads across regions and providers. A control plane that treats\nseparate pools as one avoids leaving capacity idle in one pool while another\nruns short.</p></li></ul>\n<h3 id=\"3-10-change-management\">3.10 Change management</h3>\n<ul><li><p><strong>Track a three-way version matrix:</strong> model, hardware and engine build. New</p><p>models often need a nightly or model-specific engine build on day zero.</p></li><li><p><strong>Plan rollouts for exclusive GPUs.</strong> A GPU claim is exclusive, so a rolling</p><p>update on a one-replica deployment deadlocks: the new pod cannot schedule and\nthe old one never leaves. Use a Recreate strategy or allow one unavailable,\nand budget the downtime.</p></li><li><p><strong>Ready doesn&#39;t mean routable.</strong> A gateway can lag a ready replica by a minute</p><p>or two.</p></li><li><p><strong>Test serving changes like code.</strong> Make each change a pull request and change</p><p>one thing at a time. Write the bar down first. Sweep load upward until a step\nmisses the bar, and report capacity as one number with a confidence interval.</p></li><li><p><strong>Test the test rig first.</strong> A rig bug can waste a scarce GPU window that a</p><p>few minutes on a laptop would have caught.</p></li><li><p><strong>Test on your own hardware with traffic like yours.</strong> Agent prompts are long</p><p>and mostly cached. A benchmark of short, fresh prompts measures a different\nsystem.</p></li></ul>\n<p>Promote every serving change through tiers. Each tier costs more than the one\nbefore and catches a different kind of failure, so most problems show up before\nusers see them:</p>\n<ul><li><p><strong>Offline.</strong> Validate the config and run the test rig&#39;s own checks. This takes</p><p>minutes on a laptop and no GPU.</p></li><li><p><strong>Rehearsal.</strong> Run the full test end to end against a fake gateway or a small</p><p>model, so rig bugs don&#39;t cost a GPU window.</p></li><li><p><strong>Smoke.</strong> Send a few real requests on the target hardware. Include a long</p><p>prompt, a tool call and a streamed answer, plus one request that should be\nrefused.</p></li><li><p><strong>Load with evals.</strong> Sweep concurrency up to your bar with the eval suite</p><p>running alongside, and compare against the current version on the same\ntraffic. Evals can pass one request at a time while tool calls break under\nconcurrent load, with nothing moving on the serving dashboards.</p></li><li><p><strong>Canary.</strong> Send a small share of real traffic to the new version and compare</p><p>its server-side signals with the current one: TTFT, time per output token,\nerrors, tool-call parse failures and length-limit stops. A gateway that routes\nby model name can split traffic by weight. Write the rollback trigger down\nbefore you start.</p></li><li><p><strong>Full rollout.</strong> Keep the old version ready to take traffic back until the</p><p>new one&#39;s signals stay within the bar through a peak in traffic.</p></li></ul>\n<h3 id=\"3-11-quality-and-evals\">3.11 Quality and evals</h3>\n<p>When an open model scores low, check three places before you blame the model:</p>\n<ol><li><p><strong>The harness.</strong> Is the model the only thing that changed? Run your current</p><p>  hosted model first. If it scores below a weaker model, the harness is\n  broken.</p></li><li><p><strong>The infrastructure.</strong> Did the model get a chance to answer? Timeouts,</p><p>  expired credentials and routing errors show up as model failures. Mark those\n  runs &quot;not measured.&quot;</p></li><li><p><strong>The grader.</strong> Does the check measure the outcome you care about? Grade the</p><p>  end state, not the agent&#39;s description of it, and test your graders against\n  answers people already accepted or rejected.</p></li></ol>\n<p>Also pin the data the right answer depends on and give every model the same\ncontext. Keep test markers where the model cannot read them. Run each test more\nthan once, and prefer deterministic checks over using a model as the judge. A\nbatching server can return different tokens for the same request even at\ntemperature 0, so expect some variance unless the engine runs in a slower\nbatch-invariant mode.</p>\n<h3 id=\"3-12-incidents-and-debugging\">3.12 Incidents and debugging</h3>\n<ul><li><p><strong>Expect silent failures.</strong> The costly problems often don&#39;t throw errors: a</p><p>truncated stream returns 200, a metrics API reports available, node groups\nreport ready. Part of the platform’s job is to make those failures loud.</p></li><li><p><strong>Look at the duration.</strong> A failure at a round number of seconds is a timeout.</p><p>A multiple of it, such as 239 seconds after four 60-second tries, points to\nretries.</p></li><li><p><strong>Bypass a layer.</strong> Take one hop out of the path and see whether the problem</p><p>moves.</p></li><li><p><strong>Verify status against reality.</strong> A status is one component&#39;s view of its own</p><p>job. Check the disk, the instance or the route itself.</p></li><li><p><strong>Work additively, and upstream the fix.</strong> Keep workarounds beside the</p><p>platform, labelled for teardown, rather than editing objects a controller\nowns.</p></li></ul>\n<h3 id=\"checklist-before-your-first-users\">Checklist before your first users</h3>\n<ul><li><p><input type=\"checkbox\" disabled /> Use cases defined, with a written bar for time to first token, speed per</p><p>user and error rate</p></li><li><p><input type=\"checkbox\" disabled /> Model chosen on your own evals, with your current hosted model as the</p><p>baseline</p></li><li><input type=\"checkbox\" disabled /> Engine build confirmed to load the model on your hardware</li><li><p><input type=\"checkbox\" disabled /> Memory budget covers weights, KV cache and runtime overhead, with max</p><p>output tokens and max concurrent sequences set</p></li><li><p><input type=\"checkbox\" disabled /> GPU placement verified against the driver, with a fallback instance shape</p><p>or zone</p></li><li><p><input type=\"checkbox\" disabled /> Auth policy applied before the gateway gets an address, and TLS on a real</p><p>hostname</p></li><li><input type=\"checkbox\" disabled /> Credentials that last as long as an agent session</li><li><p><input type=\"checkbox\" disabled /> Timeouts raised at every hop: route, load balancers, gateway retries,</p><p>client and helper calls</p></li><li><p><input type=\"checkbox\" disabled /> Engine metrics scraped, GPU SM and tensor activity graphed, and an alert</p><p>when metrics stop arriving</p></li><li><p><input type=\"checkbox\" disabled /> Server-side counts of tool-call parse failures, length-limit stops and</p><p>empty responses, so users aren&#39;t your first alert</p></li><li><p><input type=\"checkbox\" disabled /> A probe matrix of requests that must be refused and paths that must stay</p><p>open</p></li><li><input type=\"checkbox\" disabled /> A rollout strategy for exclusive GPUs, with downtime budgeted</li><li><input type=\"checkbox\" disabled /> Previous container logs preserved after a crash</li><li><p><input type=\"checkbox\" disabled /> Serving changes promoted through tiers, with evals under concurrent load</p><p>and a canary with a written rollback trigger</p></li><li><p><input type=\"checkbox\" disabled /> Workarounds kept beside the platform, labelled for teardown, with fixes</p><p>sent upstream</p></li></ul>\n<h2 id=\"4-the-stack\">4. The stack</h2>\n<p>Go deep on the engine. You already know Kubernetes, gateways and observability,\nso learn what changes for inference. For kernels, reading a profile is enough.\nMost inference engineering happens higher up the stack.</p>\n<h3 id=\"tools-by-layer\">Tools by layer</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>Layer</th><th>Tools to know</th><th>Master first</th></tr></thead><tbody><tr><td>GPUs on Kubernetes</td><td><a href=\"https://github.com/NVIDIA/gpu-operator\" rel=\"nofollow ugc noopener\">NVIDIA GPU Operator</a>, <a href=\"https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/\" rel=\"nofollow ugc noopener\">Dynamic Resource Allocation</a> and its <a href=\"https://github.com/NVIDIA/k8s-dra-driver-gpu\" rel=\"nofollow ugc noopener\">NVIDIA driver</a>, <a href=\"https://github.com/NVIDIA/dcgm-exporter\" rel=\"nofollow ugc noopener\">DCGM exporter</a></td><td>How GPUs are advertised, requested and placed</td></tr><tr><td>Kernels and libraries</td><td><a href=\"https://developer.nvidia.com/cuda-toolkit\" rel=\"nofollow ugc noopener\">CUDA</a>, <a href=\"https://pytorch.org\" rel=\"nofollow ugc noopener\">PyTorch</a>, <a href=\"https://github.com/triton-lang/triton\" rel=\"nofollow ugc noopener\">Triton</a>, <a href=\"https://github.com/Dao-AILab/flash-attention\" rel=\"nofollow ugc noopener\">FlashAttention</a>, torch.compile</td><td>Kernel basics, kernel fusion, and how to read a profile</td></tr><tr><td>Inference engines</td><td><a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM</a>, <a href=\"https://github.com/sgl-project/sglang\" rel=\"nofollow ugc noopener\">SGLang</a>, <a href=\"https://github.com/NVIDIA/TensorRT-LLM\" rel=\"nofollow ugc noopener\">TensorRT-LLM</a>; <a href=\"https://github.com/ollama/ollama\" rel=\"nofollow ugc noopener\">Ollama</a> and <a href=\"https://github.com/ggml-org/llama.cpp\" rel=\"nofollow ugc noopener\">llama.cpp</a> for local work</td><td>vLLM&#39;s core flags: <code>--gpu-memory-utilization</code>, <code>--max-num-seqs</code>, <code>--max-model-len</code>, tensor parallel size, quantization, prefix caching, chunked prefill</td></tr><tr><td>Distributed serving</td><td><a href=\"https://github.com/ai-dynamo/dynamo\" rel=\"nofollow ugc noopener\">NVIDIA Dynamo</a>, <a href=\"https://github.com/llm-d/llm-d\" rel=\"nofollow ugc noopener\">llm-d</a>, <a href=\"https://github.com/kserve/kserve\" rel=\"nofollow ugc noopener\">KServe</a>, <a href=\"https://github.com/kubernetes-sigs/lws\" rel=\"nofollow ugc noopener\">LeaderWorkerSet</a></td><td>Multi-GPU and multi-node layouts, and prefill and decode split</td></tr><tr><td>Gateway and routing</td><td><a href=\"https://github.com/envoyproxy/gateway\" rel=\"nofollow ugc noopener\">Envoy Gateway</a>, <a href=\"https://theagentrouter.ai\" rel=\"nofollow ugc noopener\">Agent Router</a> (formerly Envoy AI Gateway), <a href=\"https://github.com/kubernetes-sigs/gateway-api-inference-extension\" rel=\"nofollow ugc noopener\">Gateway API Inference Extension</a></td><td>Streaming, timeouts, auth policies, model-name and cache-aware routing</td></tr><tr><td>Control plane</td><td><a href=\"https://github.com/modelplaneai/modelplane\" rel=\"nofollow ugc noopener\">Modelplane</a>, built on <a href=\"https://github.com/crossplane/crossplane\" rel=\"nofollow ugc noopener\">Crossplane</a></td><td>Declaring clusters, hardware classes, deployments and caches across environments</td></tr><tr><td>Observability</td><td><a href=\"https://prometheus.io\" rel=\"nofollow ugc noopener\">Prometheus</a>, <a href=\"https://github.com/grafana/grafana\" rel=\"nofollow ugc noopener\">Grafana</a>, <a href=\"https://opentelemetry.io\" rel=\"nofollow ugc noopener\">OpenTelemetry</a>, <a href=\"https://github.com/grafana/loki\" rel=\"nofollow ugc noopener\">Loki</a>, <a href=\"https://github.com/slok/sloth\" rel=\"nofollow ugc noopener\">Sloth</a> or <a href=\"https://github.com/pyrra-dev/pyrra\" rel=\"nofollow ugc noopener\">Pyrra</a></td><td>Engine metrics, real GPU activity, and SLOs on user-facing latency</td></tr><tr><td>Benchmarking</td><td><a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM&#39;s benchmark tools</a>, <a href=\"https://github.com/ai-dynamo/aiperf\" rel=\"nofollow ugc noopener\">NVIDIA AIPerf</a>, <a href=\"https://github.com/vllm-project/guidellm\" rel=\"nofollow ugc noopener\">GuideLLM</a>, <a href=\"https://inferencex.semianalysis.com/agentx/methodology\" rel=\"nofollow ugc noopener\">SemiAnalysis AgentX</a> traces</td><td>Concurrency sweeps and latency percentiles under realistic prompts</td></tr><tr><td>Evals</td><td><a href=\"https://inspect.aisi.org.uk\" rel=\"nofollow ugc noopener\">Inspect</a> from the UK AI Security Institute; Hugging Face upskill for testing skills across models</td><td>Deterministic graders on the end state of a task</td></tr><tr><td>Models</td><td><a href=\"https://huggingface.co/models\" rel=\"nofollow ugc noopener\">Hugging Face Hub</a>, <a href=\"https://github.com/huggingface/safetensors\" rel=\"nofollow ugc noopener\">safetensors</a> checkpoints</td><td>Reading a model config for layers, heads and context, and checking real file size</td></tr></tbody></table></div>\n<h3 id=\"choosing-an-engine\">Choosing an engine</h3>\n<p>The engine controls batching, memory management, kernel selection and API\ncompatibility. It often matters more to cost and throughput than a hardware\nupgrade.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Engine</th><th>Why teams pick it</th><th>What it costs to operate</th></tr></thead><tbody><tr><td><a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM</a></td><td>The broadest model coverage, plus OpenAI and Anthropic APIs out of the box, so it is the usual first deployment. Hosted by the PyTorch Foundation.</td><td>Frequent releases with new flags. Pin versions and re-benchmark on every upgrade.</td></tr><tr><td><a href=\"https://github.com/sgl-project/sglang\" rel=\"nofollow ugc noopener\">SGLang</a></td><td>A radix-tree prefix cache and a scheduler that favors requests sharing a cached prefix, which suits agents and long multi-turn sessions. Upbound measured its biggest single gain here (Project Champagne, Part 5, 2026).</td><td>A smaller community, and new models may need nightly builds before a tagged release works.</td></tr><tr><td><a href=\"https://github.com/NVIDIA/TensorRT-LLM\" rel=\"nofollow ugc noopener\">TensorRT-LLM</a></td><td>Kernels tuned for NVIDIA hardware, including FP4 on Blackwell</td><td>NVIDIA hardware only, which ties your serving stack to one vendor.</td></tr><tr><td><a href=\"https://github.com/ai-dynamo/dynamo\" rel=\"nofollow ugc noopener\">NVIDIA Dynamo</a></td><td>A layer above engines that coordinates many nodes, including split prefill and decode</td><td>Another distributed system to run, so use it only at multi-node scale.</td></tr><tr><td><a href=\"https://github.com/ollama/ollama\" rel=\"nofollow ugc noopener\">Ollama</a>, <a href=\"https://github.com/ggml-org/llama.cpp\" rel=\"nofollow ugc noopener\">llama.cpp</a></td><td>Local serving for learning and prototyping</td><td>Not designed for many concurrent users</td></tr></tbody></table></div>\n<p>Hugging Face&#39;s TGI was a common choice and is now in maintenance mode, so check\nrelease cadence, maintainers and foundation backing before you commit to one.\nEngines add new flags every few weeks. Re-run your benchmarks when you upgrade,\nrather than trusting last quarter&#39;s numbers.</p>\n<p>Upbound&#39;s own comparison in <a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team\" rel=\"nofollow ugc noopener\">Project Champagne, Part\n5</a>\nshows this. The team was serving a coding model to about 10 engineers on two\n8-GPU nodes and tested how to grow to a team of 15. On the same hardware and the\nsame agent traffic, switching from vLLM to SGLang was the largest single gain it\nmeasured. Adding SGLang&#39;s host-memory cache on top didn&#39;t add a measurable gain,\nbecause the KV cache never filled.</p>\n<h3 id=\"where-modelplane-fits\">Where Modelplane fits</h3>\n<p><a href=\"https://github.com/modelplaneai/modelplane\" rel=\"nofollow ugc noopener\">Modelplane</a> is an open source\nproject from Upbound, built on\n<a href=\"https://github.com/crossplane/crossplane\" rel=\"nofollow ugc noopener\">Crossplane</a> and released under the\nApache 2.0 license. It orchestrates models, inference engines and AI\ninfrastructure across clouds, neoclouds and your own GPUs, with Kubernetes-style\ndeclarative APIs. It provisions inference clusters or uses ones you bring.\nDocumentation is at <a href=\"https://docs.modelplane.ai\" rel=\"nofollow ugc noopener\">docs.modelplane.ai</a>.</p>\n<p>Modelplane splits the work into three layers of ownership. You can use the same\nsplit as a mental model even with different tools.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Who</th><th>Owns</th><th>Modelplane resources</th></tr></thead><tbody><tr><td>Platform team</td><td>The fleet: clusters, hardware classes and the gateway</td><td>InferenceCluster, InferenceClass, InferenceGateway</td></tr><tr><td>Model operator, or inference engineer</td><td>The models: deployments, services and weight caches</td><td>ModelDeployment, ModelService, ModelCache</td></tr><tr><td>Modelplane</td><td>Everything in between, composed from the two above</td><td>ModelEndpoint, ModelReplica</td></tr></tbody></table></div>\n<p>The hierarchy mirrors core Kubernetes one scope up. ModelDeployment →\nModelReplica → ModelService → ModelEndpoint parallels Deployment → Pod → Service\n→ Endpoint, across a fleet instead of within one cluster. You bring the engine\ncontainer and its flags, and Modelplane shapes the topology around them. People\nand agents never create any of these. They call the InferenceGateway by model\nname.</p>\n<p>Modelplane maps vLLM and SGLang metrics onto common names out of the box, and a\nMetricMapping adds any other engine, so different engines are judged on the same\nsignals.</p>\n<p>For a platform engineer, the second layer is where the new learning is. Upbound\nfound the same thing when it started self-hosting in <a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence\" rel=\"nofollow ugc noopener\">Project Champagne, Part\n1</a>:\nthe cluster and gateway work was familiar, and the model operator layer held the\nunknowns.</p>\n<h3 id=\"a-suggested-order-to-learn-the-stack\">A suggested order to learn the stack</h3>\n<ol><li><p>Run vLLM on one GPU behind its OpenAI-compatible API, and read its metrics</p><p>  endpoint.</p></li><li><p>Deploy it on Kubernetes with the NVIDIA GPU Operator, and place it with</p><p>  device selectors.</p></li><li><p>Put a gateway in front of it with TLS, auth and timeouts sized for long</p><p>  streams.</p></li><li><p>Scrape engine and GPU metrics into Prometheus and build the core panels from</p><p>  Section 3.8.</p></li><li><p>Benchmark it with a concurrency sweep, and run a small eval suite alongside</p><p>  it.</p></li><li>Serve the same model on SGLang and compare on your own traffic.</li><li><p>Split a larger model across GPUs with tensor and pipeline parallelism, then</p><p>  try a distributed serving layer.</p></li><li><p>Learn enough CUDA and profiling to understand why a kernel or a setting made</p><p>  the difference.</p></li></ol>\n<h2 id=\"5-learn-from-teams-that-share-their-work\">5. Learn from teams that share their work</h2>\n<p>Much of what you need to know about running inference comes from teams that\nwrote up what broke. Learn to read those write-ups critically, and write your\nown (Section 6).</p>\n<h3 id=\"how-to-read-a-case-study\">How to read a case study</h3>\n<ul><li><p><strong>Find the setup.</strong> Model, hardware, engine version and traffic shape. Note</p><p>the date, because numbers age fast.</p></li><li><strong>Find the bar.</strong> What counted as good enough, and who decided.</li><li><p><strong>Follow the failures.</strong> What broke, how the team found out, and how long it</p><p>took to detect.</p></li><li><p><strong>Check the method.</strong> Did they change one thing at a time and measure it on</p><p>their own traffic?</p></li><li><p><strong>Separate what transfers.</strong> A timeout lesson applies everywhere, but a</p><p>specific flag value only holds for that engine release on that hardware.</p></li></ul>\n<h4 id=\"case-study-project-champagne-2026\">Case study: Project Champagne (2026)</h4>\n<p>Upbound, the company behind Crossplane and Modelplane, self-hosted chat, coding\nagents and a CVE-fixing build agent on open models with Modelplane, across AWS\nand Nebius GPUs. The team published what broke in a five-part series.\nHighlights, with numbers as of 2026:</p>\n<ul><li><p><strong>Serving was the smaller problem.</strong> vLLM and scheduling worked. TLS,</p><p>identity, timeouts, storage and observability took the time. (Part 2)</p></li><li><p><strong>Quota did not mean capacity.</strong> Approved quota for 768 vCPUs of H100</p><p>instances never produced one, and the planned GPU arrived about half the time.\nOne model ended up split across two nodes in two zones. (Part 2)</p></li><li><p><strong>Fitting at startup wasn&#39;t enough.</strong> A chat model that fit cleanly ran out of</p><p>memory nine times in four days. Lowering the memory fraction from 0.92 to\n0.88, enabling expandable segments and capping concurrent sequences at 16\nfreed 4.6 GiB. (Part 2)</p></li><li><p><strong>The expensive failures were silent.</strong> A truncated stream returned 200, a</p><p>metrics API reported available with no data, and monitoring sat pending for\nsix days. (Part 2)</p></li><li><p><strong>Timeouts stacked up.</strong> A 15-second route default, then a 60-second load</p><p>balancer idle timeout, cut off answers that were still streaming. (Parts 2 and\n3)</p></li><li><p><strong>Agents were the hardest clients.</strong> Hour-long sessions ran past the 60-minute</p><p>lifetime of their tokens, and a reasoning model thought for more than a minute\nbefore its first visible token. (Parts 2 and 3)</p></li><li><p><strong>Low scores were often not the model’s fault.</strong> A fake shell in the test</p><p>harness scored Claude 0. A 60-second idle timeout failed a model that\ncompleted every run once the limit was raised. (Part 3)</p></li><li><p><strong>Slowness had several causes.</strong> An engineer’s “wow, is it slow” traced back</p><p>to hidden maximum-effort reasoning while every dashboard looked fine. One GPU\nshowed 99% <code>GPU-Util</code> with tensor cores busy about 10% of the time. (Part 4)</p></li><li><p><strong>Load depended on the client.</strong> Agent prompts had a median of 59K tokens,</p><p>about two-thirds hit the prefix cache, and short-prompt sessions kept more\nrequests in flight by launching subagents. (Part 5)</p></li><li><p><strong>The engine mattered more than expected.</strong> On the same two 8-GPU nodes,</p><p>moving from vLLM to SGLang was the largest single gain. A 128K context cap\nonly looked faster because it rejected long conversations. (Part 5)</p></li></ul>\n<h3 id=\"read-the-full-series\">Read the full series</h3>\n<p>Each post on the Upbound blog tells these stories in detail, with the configs,\nqueries and numbers behind them:</p>\n<ul><li><p>[**Part 1: What it takes to own your</p><p>intelligence.**](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence</a>)\nWhy Upbound self-hosts inference, and how the use cases picked the models and\nhardware.</p></li><li><p>[**Part 2: Finding the</p><p>corks.**](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks</a>)\nThe inference lifecycle stage by stage, from pick and place to change and\nfail.</p></li><li><p>[**Part 3: The build security</p><p>agent.**](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent</a>)\nHow to tell whether an open model can do a real job as well as a hosted one.</p></li><li><p>[**Part 4: Candling the</p><p>bottles.**](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles</a>)\nWhat the team could see at each step, what it missed, and how it plans to\nalert.</p></li><li><p>[**Part 5: From pilot to</p><p>team.**](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team</a>)\nA repeatable way to test serving changes and size capacity for a growing team.</p></li></ul>\n<h2 id=\"6-making-the-career-move\">6. Making the career move</h2>\n<p>You don&#39;t need to start over. Add inference concepts to the platform skills you\nhave, and show the work in public. The field is young, so newcomers can catch up\nfast.</p>\n<h3 id=\"pick-a-direction\">Pick a direction</h3>\n<p>Pick the direction closest to the work you already do. Most platform engineers\nstart in the first two and grow toward the third.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Direction</th><th>Focus</th><th>Good fit if you</th></tr></thead><tbody><tr><td>Inference platform</td><td>Clusters, GPU scheduling, gateways, autoscaling, multi-cloud capacity</td><td>Enjoy Kubernetes and distributed systems</td></tr><tr><td>Inference reliability</td><td>SLOs, observability, capacity planning, rollouts, incident response</td><td>Have carried a pager and run SRE practices</td></tr><tr><td>Runtime performance</td><td>Engine tuning, quantization, speculative decoding, parallelism, kernels</td><td>Like profiling and working close to the hardware</td></tr></tbody></table></div>\n<h3 id=\"follow-a-learning-path\">Follow a learning path</h3>\n<p>Vishakha Sadhwani&#39;s <em>AI Inference Engineering Roadmap 2026</em> is a thorough\n13-stage map of the field. Coming from platform engineering, you can move\nthrough it faster, because you already own much of the infrastructure half. A\nshorter path:</p>\n<ol><li><p><strong>Learn the model, not just the container.</strong> Transformer basics, attention</p><p>  and the KV cache, dense versus MoE models, and how to read a model config.\n  The goal is to estimate the memory any model needs in five minutes.</p></li><li><p><strong>Learn one engine deeply.</strong> Pick vLLM, learn what each core flag trades</p><p>  away, and measure time to first token and per-token latency yourself.</p></li><li><p><strong>Translate your platform skills to GPUs.</strong> Device allocation, gateways that</p><p>  hold long streams, autoscaling on inference signals, and observability for\n  engines.</p></li><li><p><strong>Learn to judge quality and cost.</strong> Build an eval, run a capacity sweep and</p><p>  calculate cost per million tokens. This is where platform engineers most\n  often have a gap, and where the role adds the most value.</p></li></ol>\n<h3 id=\"build-projects-that-prove-it\">Build projects that prove it</h3>\n<p>Each project below fits on a rented GPU or two, and each produces a repo with\nreal numbers. Shut instances down between runs, and use spot capacity for batch\nexperiments.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Project</th><th>What it proves</th><th>What to publish</th></tr></thead><tbody><tr><td>Serve an 8B model with vLLM on one GPU. Sweep 1, 10 and 50 concurrent users.</td><td>You can measure TTFT, time per output token and throughput accurately</td><td>Configs, the load script, charts and cost per million tokens</td></tr><tr><td>Predict how many 32K-token sessions fit on your GPU, then test it</td><td>You understand weights, KV cache and runtime overhead</td><td>Your prediction, the result and why they differ</td></tr><tr><td>Compare FP16 and FP8, and a 4-bit format if you can</td><td>You can trade quality against speed with evidence</td><td>Eval scores next to latency and throughput</td></tr><tr><td>Run the same prompts with shared content first, then last</td><td>You understand prefix caching and prompt structure</td><td>Cache hit rate and TTFT for each</td></tr><tr><td>Deploy on Kubernetes with the GPU Operator, a gateway with TLS and auth, autoscaling on queue depth, and dashboards</td><td>You can run inference as a platform</td><td>Manifests or compositions, and dashboard definitions</td></tr><tr><td>Compare vLLM and SGLang on agent-like traffic, one change at a time</td><td>You make engine choices with data</td><td>The bar, the method and results with intervals</td></tr><tr><td>Build an eval harness for one agent task with deterministic graders</td><td>You can tell a model failure from a harness failure</td><td>Tasks, graders and negative controls</td></tr><tr><td>Reproduce a failure: a timeout, an out-of-memory crash or a stuck rollout</td><td>You can operate it, not just deploy it</td><td>A short postmortem and runbook</td></tr></tbody></table></div>\n<h3 id=\"show-your-work-in-public\">Show your work in public</h3>\n<p>Many inference roles fill through networks before they reach a careers board. In\nAugust 2026, <a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">Zeoli\nfound</a> one\nserving company valued at \\$17.5B that didn&#39;t list any dedicated inference\nperformance openings. Public work is how people outside those networks get\nfound.</p>\n<ul><li><p><strong>Write READMEs like lab reports.</strong> State the question, the setup (model, GPU,</p><p>engine version and flags), the method, the results with numbers, what\nsurprised you, and how to reproduce it.</p></li><li><p><strong>Publish what failed.</strong> A clear account of a failure shows more skill than a</p><p>clean demo.</p></li><li><p><strong>Pin versions and show cost.</strong> Results without an engine version and a price</p><p>are hard to trust and harder to reuse.</p></li><li><p><strong>Write and speak about it.</strong> Turn each project into a short post. Submit</p><p>talks to meetups, CNCF events and community live streams.</p></li></ul>\n<h3 id=\"contribute-to-open-source\">Contribute to open source</h3>\n<p>Much of what inference projects still need is platform work. Upbound&#39;s own\nupstream list from Project Champagne reads like a platform backlog: configurable\nload balancers, certificate issuance, Recreate rollouts for exclusive devices,\nan honest Ready signal, engine metrics scraped out of the box, and auth that\nsuits long-running agents.</p>\n<p>Projects to follow include <a href=\"https://github.com/vllm-project/vllm\" rel=\"nofollow ugc noopener\">vLLM</a>,\n<a href=\"https://github.com/sgl-project/sglang\" rel=\"nofollow ugc noopener\">SGLang</a>,\n<a href=\"https://github.com/llm-d/llm-d\" rel=\"nofollow ugc noopener\">llm-d</a>,\n<a href=\"https://github.com/kserve/kserve\" rel=\"nofollow ugc noopener\">KServe</a>, <a href=\"https://github.com/ai-dynamo/dynamo\" rel=\"nofollow ugc noopener\">NVIDIA\nDynamo</a>, <a href=\"https://theagentrouter.ai\" rel=\"nofollow ugc noopener\">Agent\nRouter</a> (formerly Envoy AI Gateway), the <a href=\"https://github.com/kubernetes-sigs/gateway-api-inference-extension\" rel=\"nofollow ugc noopener\">Gateway API\nInference\nExtension</a>,\n<a href=\"https://github.com/modelplaneai/modelplane\" rel=\"nofollow ugc noopener\">Modelplane</a> and\n<a href=\"https://github.com/crossplane/crossplane\" rel=\"nofollow ugc noopener\">Crossplane</a>. Good first\ncontributions:</p>\n<ul><li><p><strong>Run it and report what breaks,</strong> with a reproduction and numbers. This is</p><p>how Champagne contributed most.</p></li><li><p><strong>Write operator documentation:</strong> deployment, tuning and troubleshooting</p><p>guides drawn from your own runs.</p></li><li><strong>Add metrics, dashboards and alerts</strong> that make silent failures loud.</li><li><strong>Improve manifests, Helm charts and compositions</strong> for real environments.</li><li><strong>Contribute benchmark recipes and tests</strong> for the hardware you have.</li></ul>\n<p>Read each project&#39;s contributing guide, start with a small change, and comment\non an issue before taking on a large one.</p>\n<h3 id=\"ongoing-learning\">Ongoing learning</h3>\n<p>Engines add flags every few weeks, and research reaches production within\nmonths.</p>\n<ul><li><p><strong>Read release notes</strong> for the engine you run, every release, and re-run your</p><p>benchmark when you upgrade.</p></li><li><p><strong>Read the core papers:</strong></p><p><a href=\"https://www.usenix.org/conference/osdi22/presentation/yu\" rel=\"nofollow ugc noopener\">Orca</a> on continuous\nbatching, <a href=\"https://arxiv.org/abs/2309.06180\" rel=\"nofollow ugc noopener\">PagedAttention</a>,\n<a href=\"https://arxiv.org/abs/2401.09670\" rel=\"nofollow ugc noopener\">DistServe</a> on disaggregation, the original\n<a href=\"https://arxiv.org/abs/2211.17192\" rel=\"nofollow ugc noopener\">speculative decoding</a> paper, and\n<a href=\"https://arxiv.org/abs/2205.14135\" rel=\"nofollow ugc noopener\">FlashAttention</a>.</p></li><li><p>*<em>Read Philip Kiely&#39;s [</em>Inference</p><p>Engineering<em>](<a href=\"https://www.baseten.co/inference-engineering/\" rel=\"nofollow ugc noopener\">https://www.baseten.co/inference-engineering/</a>).</em>* The e-book is\nfree from Baseten.</p></li><li><p><strong>Follow practitioners.</strong> Newsletters such as [The Pragmatic</p><p>Engineer](<a href=\"https://newsletter.pragmaticengineer.com\" rel=\"nofollow ugc noopener\">https://newsletter.pragmaticengineer.com</a>),\n<a href=\"https://blog.bytebytego.com\" rel=\"nofollow ugc noopener\">ByteByteGo</a> and <a href=\"https://www.datagravity.dev\" rel=\"nofollow ugc noopener\">Data\nGravity</a> cover the field well. Join the community\nchannels for the engines you use.</p></li><li><p><strong>Learn evals.</strong> Whether an open model can replace a hosted one is decided by</p><p>evals, and few engineers can build good ones. Start with Anthropic&#39;s\n<a href=\"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\" rel=\"nofollow ugc noopener\">*Demystifying evals for AI\nagents*</a>\nand Hamel Husain and Shreya Shankar&#39;s <a href=\"https://hamel.dev/blog/posts/evals-faq/\" rel=\"nofollow ugc noopener\">evals\nFAQ</a>.</p></li><li><p><strong>Learn the economics.</strong> Be able to explain cost per million tokens and cost</p><p>per user session to a finance partner.</p></li></ul>\n<h3 id=\"find-the-roles\">Find the roles</h3>\n<p>Don&#39;t expect a consistent job title yet. The same work shows up as inference\nengineer, performance engineer, model serving engineer or ML infrastructure\nengineer, and sometimes as AI platform engineer, so search for all of them. Pay\nreflects how scarce the skills are: in August 2026, <a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">Zeoli\nfound</a> an\nAnthropic inference performance role posted at \\$350,000 to \\$850,000, and an\nOpenAI equivalent at \\$295,000 to \\$555,000.</p>\n<p>You may not need to change jobs to get started, though. If your company is\nplanning to self-host its first model, put your hand up. Your team will probably\nend up responsible for that project anyway, and running it is the most direct\nexperience you can get.</p>\n<h3 id=\"prepare-for-interviews\">Prepare for interviews</h3>\n<p>Practice answering these. Each one points to the section of this guide that\ncovers it. Several are adapted from Lamhot Siagian&#39;s interview pocket notes on\nefficient inference (Section 7).</p>\n<ul><li><p><strong>Prefill and decode.</strong> Why is decode usually limited by memory bandwidth</p><p>rather than compute, and what changes at long context? (Section 2)</p></li><li><p><strong>Hardware trends.</strong> If compute keeps growing faster than memory bandwidth,</p><p>will inference soon become compute-bound? (Section 2)</p></li><li><p><strong>Memory math.</strong> How much memory do a model and its KV cache need at a given</p><p>context length and concurrency? (Section 2)</p></li><li><strong>Goodput.</strong> What is goodput, and why plan capacity with it? (Section 2)</li><li><p><strong>Prefix caching.</strong> Which applications benefit most from prefix caching?</p><p>(Section 2)</p></li><li><p><strong>Precision.</strong> You need to serve a 70B model on two 80 GB GPUs. What precision</p><p>would you choose, and why? (Sections 2 and 3.4)</p></li><li><p><strong>Quantization.</strong> When does weight-only quantization speed things up, and when</p><p>does it not? (Section 3.4)</p></li><li><p><strong>Speculative decoding.</strong> When would you not use speculative decoding?</p><p>(Section 3.4)</p></li><li><p><strong>Chunked prefill.</strong> How do long prompts slow other users&#39; streams, and what</p><p>does chunked prefill change? (Section 3.4)</p></li><li><p><strong>Parallelism.</strong> Which parallelism strategy would you pick for a given model</p><p>and GPU type, and when does tensor parallelism beat pipeline parallelism?\n(Sections 3.4 and 3.5)</p></li><li><p><strong>Profiling.</strong> How would you tell whether a workload is limited by compute or</p><p>by memory bandwidth? (Section 3.8)</p></li><li><p><strong>Debugging &quot;slow.&quot;</strong> p99 time to first token just spiked. How do you tell</p><p>queueing, hidden reasoning and slow generation apart? (Section 3.8)</p></li><li><p><strong>Cost.</strong> What does a million tokens cost on your GPUs, and how does that</p><p>compare with a hosted API? (Section 2)</p></li><li><p><strong>Capacity planning.</strong> Your user count will grow 10x in six months. How do you</p><p>plan capacity? (Section 3.10)</p></li><li><p><strong>Rollouts and SLOs.</strong> How would you roll out a new model version safely, and</p><p>what SLO would you set for the service? (Sections 3.8 and 3.10)</p></li><li><p><strong>Failure stories.</strong> Describe one failure with real numbers. Cover what broke</p><p>and how you found it, then what you changed.</p></li></ul>\n<h2 id=\"7-sources-and-further-reading\">7. Sources and further reading</h2>\n<h3 id=\"upbound-and-modelplane\">Upbound and Modelplane</h3>\n<ul><li><p>Upbound, <em>Drinking Our Own Champagne</em> series on the Upbound blog: [Part 1:</p><p>What it takes to own your\nintelligence](<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence\" rel=\"nofollow ugc noopener\">https://www.upbound.io/blog/drinking-our-own-champagne-what-it-takes-to-own-your-intelligence</a>);\n<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-2-finding-the-corks\" rel=\"nofollow ugc noopener\">Part 2: Finding the\ncorks</a>;\n<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-3-use-case-1-the-build-security-agent\" rel=\"nofollow ugc noopener\">Part 3: Use case #1, the build security\nagent</a>;\n<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-4-candling-the-bottles\" rel=\"nofollow ugc noopener\">Part 4: Candling the\nbottles</a>;\n<a href=\"https://www.upbound.io/blog/drinking-our-own-champagne-part-5-what-does-it-take-to-grow-a-self-hosted-coding-model-from-pilot-to-team\" rel=\"nofollow ugc noopener\">Part 5: What does it take to grow a self-hosted coding model from pilot to\nteam?</a></p></li><li><p>Viktor Farcic, *LLM Inference Explained: 12 Concepts You Actually Need to</p><p>Know*.</p></li><li><p><a href=\"https://github.com/modelplaneai/modelplane\" rel=\"nofollow ugc noopener\">Modelplane on GitHub</a> and</p><p><a href=\"https://docs.modelplane.ai\" rel=\"nofollow ugc noopener\">Modelplane documentation</a>.</p></li></ul>\n<h3 id=\"books-and-articles\">Books and articles</h3>\n<ul><li><p>Philip Kiely, [*Inference</p><p>Engineering*](<a href=\"https://www.baseten.co/inference-engineering/\" rel=\"nofollow ugc noopener\">https://www.baseten.co/inference-engineering/</a>), Baseten Books,</p>\n<ol start=\"2026\"><li>Free as an e-book.</li></ol></li><li><p>Gergely Orosz &amp; Philip Kiely, [*What is inference engineering?</p><p>Deepdive*](<a href=\"https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering\" rel=\"nofollow ugc noopener\">https://newsletter.pragmaticengineer.com/p/what-is-inference-engineering</a>),\nThe Pragmatic Engineer, March 31, 2026.</p></li><li><p>ByteByteGo, [*A Guide to AI Inference</p><p>Engineering*](<a href=\"https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering\" rel=\"nofollow ugc noopener\">https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering</a>),\nJune 15, 2026.</p></li><li><p>Chris Zeoli, [*The Rise of Inference</p><p>Engineering*](<a href=\"https://www.datagravity.dev/p/the-rise-of-inference-engineering\" rel=\"nofollow ugc noopener\">https://www.datagravity.dev/p/the-rise-of-inference-engineering</a>),\nData Gravity, August 13, 2026.</p></li><li><p>Spheron, [*What Is Inference Engineering? The 2026 GPU Cloud</p><p>Guide*](<a href=\"https://www.spheron.network/blog/inference-engineering-guide-2026/\" rel=\"nofollow ugc noopener\">https://www.spheron.network/blog/inference-engineering-guide-2026/</a>),\nupdated October 19, 2026.</p></li><li><p>Vishakha Sadhwani, [*AI Inference Engineering Roadmap 2026: Complete Learning</p><p>Path*.](<a href=\"https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap\" rel=\"nofollow ugc noopener\">https://www.tech5ense.com/p/the-ai-inference-engineering-roadmap</a>)</p></li><li><p>Charity Majors, Liz Fong-Jones and George Miranda, [*Observability</p><p>Engineering*](<a href=\"https://www.oreilly.com/library/view/observability-engineering/9781492076438/\" rel=\"nofollow ugc noopener\">https://www.oreilly.com/library/view/observability-engineering/9781492076438/</a>),\nO&#39;Reilly.</p></li><li><p>Lamhot Siagian, Efficient LLM Inference: Interview Pocket Notes, second</p><p>edition, 2026.</p></li></ul>\n<h3 id=\"evals-and-benchmarks\">Evals and benchmarks</h3>\n<ul><li><p>Anthropic, [*Demystifying evals for AI</p><p>agents*](<a href=\"https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents\" rel=\"nofollow ugc noopener\">https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents</a>),\nJanuary 2026.</p></li><li><p>Hamel Husain and Shreya Shankar, [*AI Evals</p><p>FAQ*](<a href=\"https://hamel.dev/blog/posts/evals-faq/\" rel=\"nofollow ugc noopener\">https://hamel.dev/blog/posts/evals-faq/</a>).</p></li><li>UK AI Security Institute, <a href=\"https://inspect.aisi.org.uk\" rel=\"nofollow ugc noopener\">Inspect</a>.</li><li>Hugging Face, <a href=\"https://github.com/huggingface/upskill\" rel=\"nofollow ugc noopener\">upskill</a>.</li><li><p>SemiAnalysis, [AgentX methodology and</p><p>datasets](<a href=\"https://inferencex.semianalysis.com/agentx/methodology\" rel=\"nofollow ugc noopener\">https://inferencex.semianalysis.com/agentx/methodology</a>).</p></li></ul>\n<h3 id=\"papers\">Papers</h3>\n<ul><li><p>Yu et al., [Orca: A Distributed Serving System for Transformer-Based</p><p>Generative Models](<a href=\"https://www.usenix.org/conference/osdi22/presentation/yu\" rel=\"nofollow ugc noopener\">https://www.usenix.org/conference/osdi22/presentation/yu</a>),\nOSDI 2022.</p></li><li><p>Kwon et al., [Efficient Memory Management for Large Language Model Serving</p><p>with PagedAttention](<a href=\"https://arxiv.org/abs/2309.06180\" rel=\"nofollow ugc noopener\">https://arxiv.org/abs/2309.06180</a>), SOSP 2023.</p></li><li><p>Zhong et al., [DistServe: Disaggregating Prefill and Decoding for</p><p>Goodput-optimized Large Language Model\nServing](<a href=\"https://arxiv.org/abs/2401.09670\" rel=\"nofollow ugc noopener\">https://arxiv.org/abs/2401.09670</a>), OSDI 2024.</p></li><li><p>Leviathan et al., [Fast Inference from Transformers via Speculative</p><p>Decoding](<a href=\"https://arxiv.org/abs/2211.17192\" rel=\"nofollow ugc noopener\">https://arxiv.org/abs/2211.17192</a>), ICML 2023.</p></li><li><p>Dao et al., <a href=\"https://arxiv.org/abs/2205.14135\" rel=\"nofollow ugc noopener\">FlashAttention</a>, 2022, and Shah</p><p>et al., <a href=\"https://arxiv.org/abs/2407.08608\" rel=\"nofollow ugc noopener\">FlashAttention-3</a>, 2024.</p></li><li><p>Shoeybi et al., [Megatron-LM: Training Multi-Billion Parameter Language Models</p><p>Using Model Parallelism](<a href=\"https://arxiv.org/abs/1909.08053\" rel=\"nofollow ugc noopener\">https://arxiv.org/abs/1909.08053</a>), 2019.</p></li><li><p>Shazeer et al., [Outrageously Large Neural Networks: The Sparsely-Gated</p><p>Mixture-of-Experts Layer](<a href=\"https://arxiv.org/abs/1701.06538\" rel=\"nofollow ugc noopener\">https://arxiv.org/abs/1701.06538</a>), 2017.</p></li></ul>","headings":[{"level":1,"text":"An Open Field Guide to Inference Engineering","id":"an-open-field-guide-to-inference-engineering"},{"level":2,"text":"1. Why now?","id":"1-why-now"},{"level":3,"text":"Skills that carry over","id":"skills-that-carry-over"},{"level":2,"text":"2. Inference 101","id":"2-inference-101"},{"level":3,"text":"What an inference engine is","id":"what-an-inference-engine-is"},{"level":3,"text":"The inference stack","id":"the-inference-stack"},{"level":3,"text":"Tokens","id":"tokens"},{"level":3,"text":"Weights and the KV cache","id":"weights-and-the-kv-cache"},{"level":3,"text":"Two phases, two bottlenecks","id":"two-phases-two-bottlenecks"},{"level":3,"text":"Metrics to track","id":"metrics-to-track"},{"level":3,"text":"Latency versus throughput","id":"latency-versus-throughput"},{"level":3,"text":"When to adopt each concept","id":"when-to-adopt-each-concept"},{"level":3,"text":"What agents change","id":"what-agents-change"},{"level":2,"text":"3. Running inference in production","id":"3-running-inference-in-production"},{"level":3,"text":"3.1 Start from use cases, not hardware","id":"3-1-start-from-use-cases-not-hardware"},{"level":3,"text":"3.2 Pick models deliberately","id":"3-2-pick-models-deliberately"},{"level":3,"text":"3.3 Size for runtime, not startup","id":"3-3-size-for-runtime-not-startup"},{"level":3,"text":"3.4 Know the optimization techniques","id":"3-4-know-the-optimization-techniques"},{"level":3,"text":"3.5 Get GPUs, and place them correctly","id":"3-5-get-gpus-and-place-them-correctly"},{"level":3,"text":"3.6 Gateways, routing and timeouts","id":"3-6-gateways-routing-and-timeouts"},{"level":3,"text":"3.7 Security and identity","id":"3-7-security-and-identity"},{"level":3,"text":"3.8 Observability","id":"3-8-observability"},{"level":3,"text":"3.9 Autoscaling and cold starts","id":"3-9-autoscaling-and-cold-starts"},{"level":3,"text":"3.10 Change management","id":"3-10-change-management"},{"level":3,"text":"3.11 Quality and evals","id":"3-11-quality-and-evals"},{"level":3,"text":"3.12 Incidents and debugging","id":"3-12-incidents-and-debugging"},{"level":3,"text":"Checklist before your first users","id":"checklist-before-your-first-users"},{"level":2,"text":"4. The stack","id":"4-the-stack"},{"level":3,"text":"Tools by layer","id":"tools-by-layer"},{"level":3,"text":"Choosing an engine","id":"choosing-an-engine"},{"level":3,"text":"Where Modelplane fits","id":"where-modelplane-fits"},{"level":3,"text":"A suggested order to learn the stack","id":"a-suggested-order-to-learn-the-stack"},{"level":2,"text":"5. Learn from teams that share their work","id":"5-learn-from-teams-that-share-their-work"},{"level":3,"text":"How to read a case study","id":"how-to-read-a-case-study"},{"level":3,"text":"Read the full series","id":"read-the-full-series"},{"level":2,"text":"6. Making the career move","id":"6-making-the-career-move"},{"level":3,"text":"Pick a direction","id":"pick-a-direction"},{"level":3,"text":"Follow a learning path","id":"follow-a-learning-path"},{"level":3,"text":"Build projects that prove it","id":"build-projects-that-prove-it"},{"level":3,"text":"Show your work in public","id":"show-your-work-in-public"},{"level":3,"text":"Contribute to open source","id":"contribute-to-open-source"},{"level":3,"text":"Ongoing learning","id":"ongoing-learning"},{"level":3,"text":"Find the roles","id":"find-the-roles"},{"level":3,"text":"Prepare for interviews","id":"prepare-for-interviews"},{"level":2,"text":"7. Sources and further reading","id":"7-sources-and-further-reading"},{"level":3,"text":"Upbound and Modelplane","id":"upbound-and-modelplane"},{"level":3,"text":"Books and articles","id":"books-and-articles"},{"level":3,"text":"Evals and benchmarks","id":"evals-and-benchmarks"},{"level":3,"text":"Papers","id":"papers"}]}}