How to Speed Up LLM Inference: KV Cache to Production Serving

Q: Why is LLM inference so slow?

Because generating each token requires streaming every model weight from GPU memory. A 70B-parameter model in FP16 is about 140 GB of weights, and an H100 GPU can read memory at 3.35 TB/s — so the hardware ceiling is roughly 24 tokens per second for a single user, no matter how fast the compute units are. Decode is memory-bandwidth-bound, not compute-bound. Every serious optimization — batching, quantization, speculative decoding — is a way around that bandwidth wall.

Q: What is the KV cache and why does it matter?

The KV cache stores the attention key and value vectors of every previous token so the model doesn't recompute them for each new token. Without it, generation cost would grow quadratically. It matters because it's big: Llama-3-70B needs about 320 KB of cache per token, so one 128K-token conversation consumes roughly 40 GB — more than many GPUs have. In production, KV cache memory, not compute, usually caps how many requests a GPU can serve at once.

Q: What is the difference between prefill and decode?

Prefill processes your whole prompt in one parallel pass and produces the first token — it's compute-bound and its time scales with prompt length. Decode then generates output tokens one at a time, each step reading all weights plus the KV cache from memory — it's memory-bandwidth-bound. The two phases stress hardware so differently that large deployments now run them on separate GPU pools (disaggregation).

Q: What do TTFT and TPOT (or ITL) mean?

TTFT is time to first token: queueing plus prefill — what makes a chatbot feel unresponsive before it starts typing. TPOT (time per output token, also called inter-token latency or ITL) is the pace of the stream once it starts. End-to-end latency ≈ TTFT + TPOT × (output tokens − 1). Production SLOs are set on the p99 of both, not the average, because tail latency is what users actually feel.

Q: What is continuous batching?

A scheduling technique from the Orca paper (OSDI 2022) where the server makes batching decisions at every token step instead of per request. Finished sequences leave the batch immediately and new requests join mid-flight, so GPU slots are never wasted waiting for the longest request to finish. It's the single biggest serving-side throughput win — Orca measured 36.9x over request-level batching at the same latency — and every modern engine (vLLM, SGLang, TensorRT-LLM) implements it.

Q: What is PagedAttention?

vLLM's idea (SOSP 2023) of managing KV cache like an operating system manages RAM: in small fixed-size blocks mapped through a block table, instead of one big contiguous allocation per request. Naive allocators wasted 60–80% of KV memory on reservation and fragmentation; paging cuts waste to under 4%, so far more requests fit on the same GPU. The concept is now table stakes in every serving engine.

Q: Does quantization make models dumber?

At 8-bit (INT8/FP8), degradation is typically under 1% — effectively lossless for most workloads. Good 4-bit weight quantization (GPTQ or AWQ with group size 128) usually retains roughly 95–99% of benchmark scores depending on model and method, with losses concentrated in math, code, and long-context reasoning. Below 4 bits, quality drops become real and method-dependent. One caveat: matched benchmark averages can hide per-answer 'flips,' so evaluate on your own workload, not just MMLU.

Q: Is speculative decoding lossless?

Yes — with the standard rejection-sampling verification rule, the output distribution is mathematically identical to sampling from the large model alone. A small draft model proposes several tokens; the big model verifies them in one parallel pass, accepting or resampling per the proof in Leviathan et al. (ICML 2023). Some variants (like Medusa's typical acceptance) deliberately relax exactness for speed, so check which mode your engine uses.

Q: Should I use vLLM, SGLang, or TensorRT-LLM?

vLLM is the default general-purpose choice: broadest hardware and model coverage and the largest ecosystem. SGLang shines on prefix-heavy traffic (agents, RAG, multi-turn chat with big shared system prompts) thanks to RadixAttention caching, and on DeepSeek-style MoE deployments. TensorRT-LLM buys another 10–30% on fixed NVIDIA fleets at the cost of lock-in and complexity. For local single-user work, use llama.cpp or Ollama instead. Note that Hugging Face TGI was archived in March 2026 — don't start new projects on it.

Q: How much GPU memory do I need to serve a 70B model?

Weights alone: ~140 GB in FP16, ~70 GB in FP8, ~35-40 GB in 4-bit. Then add KV cache, which scales with concurrent tokens: Llama-3-70B needs ~320 KB per token in FP16, so 50 concurrent requests at 8K context add another ~128 GB. That's why a 70B FP16 deployment typically starts at 2–4x 80 GB GPUs with tensor parallelism, and why quantizing both weights and KV cache (FP8) roughly doubles the batch you can hold.

Type a question into any AI chatbot and watch what happens. There’s a pause — half a second, maybe two — and then words start streaming out at a steady clip. Now multiply that little interaction by hundreds of millions of users a day, put a price on every token, and you have the central systems problem of serving AI: inference — running a trained model to produce output — dominates both operating cost and user-perceived latency.

Training gets the headlines, but training a model happens once; serving it happens forever. Google disclosed in 2025 that the median Gemini text prompt costs 0.24 Wh of energy — and that per-prompt energy fell 33x in a single year, mostly through model- and serving-efficiency work: quantization, distillation, mixture-of-experts, and the batching techniques this article covers. The gap between a naive serving stack and a well-optimized one is an order of magnitude in throughput and several orders of magnitude in cost per token.

This guide covers LLM inference optimization from first principles to production: why generating tokens is slow at the physics level, what the KV cache is and why it dominates GPU memory, and how batching, quantization, and speculative decoding work around the bottleneck. Then it goes inside a modern engine like vLLM, and shows how companies from Google to two-person startups deploy, monitor, and pay for all of it. If you’ve read our explainer on how large language models actually work, this is the natural next step: that article covered what happens inside the model; this one covers how to make it fast and affordable.

Background: how we got from 0.1 to 20,000 tokens per second

In 2020–2022, serving a generative transformer meant calling model.generate() in a loop from a Hugging Face transformers process behind a web server. It worked for demos and collapsed under load, for reasons that took the industry a surprisingly long time to name precisely:

Request-level batching. Servers grouped requests into a batch, ran the whole batch until every sequence finished, then started the next batch. Short requests sat idle waiting for the longest one; new arrivals waited outside. GPU utilization cratered.
Contiguous KV cache allocation. Each request pre-reserved memory for its maximum possible output length, whether it used it or not. Profiling in the vLLM paper found that only 20–38% of allocated KV memory held actual token state — the rest was reservation waste and fragmentation.
Unfused, memory-hungry attention kernels. Standard attention materialized the full N×N score matrix in GPU memory, making long contexts brutally slow.

The fixes arrived as a rapid sequence of systems papers, each attacking one bottleneck:

2022  Orca (OSDI) ─── continuous batching: schedule per token step,
      │               not per request. 36.9x throughput vs FasterTransformer.
2022  FlashAttention ─ tile attention through on-chip SRAM; never
      │               materialize the score matrix. Exact, not approximate.
2023  Speculative decoding (Google & DeepMind, independently) ───
      │               draft cheap, verify in parallel, provably lossless.
2023  vLLM / PagedAttention (SOSP) ─── manage KV cache like virtual
      │               memory pages. Waste: 60-80% → under 4%.
2024  Sarathi-Serve / DistServe / Splitwise (OSDI, ISCA) ───
      │               chunked prefill; split prefill and decode onto
      │               separate GPU pools; optimize "goodput," not tokens/s.
2025  Mooncake (FAST, Best Paper) ─── KV-cache-centric disaggregated
      │               serving in production for Kimi. Dynamo, llm-d launch.
2026  Consolidation: TGI archived; Dynamo 1.0 GA; disaggregation and
      KV-aware routing become standard features, not research topics.

The pattern worth internalizing: almost none of these wins came from faster matrix math. Orca’s execution engine was at best ~47% faster than NVIDIA’s FasterTransformer (on the 175B, 16-GPU configuration, mostly from better cross-GPU communication) and slightly slower at smaller scales — the 36.9x came from scheduling. PagedAttention’s win came from memory management. The story of inference optimization is mostly a story about feeding the GPU properly, not about the GPU itself.

The physics: why generating tokens is slow

Everything in this field follows from one asymmetry, so let’s build it from scratch.

An LLM answers you in two phases. Prefill reads your entire prompt in one parallel pass — every prompt token processed simultaneously — and produces the first output token. Decode then generates the rest one token at a time, because each token depends on the one before it. These two phases could hardly be more different as workloads:

            PREFILL                          DECODE
   ┌────────────────────────┐      ┌────────────────────────┐
   │ 2,000 prompt tokens    │      │ 1 new token per step   │
   │ processed in parallel  │      │ × 500 steps            │
   │                        │      │                        │
   │ Arithmetic: HIGH       │      │ Arithmetic: TINY       │
   │ (thousands of tokens   │      │ (all weights read from │
   │  share one weight read)│      │  memory for ONE token) │
   │                        │      │                        │
   │ → COMPUTE-BOUND        │      │ → MEMORY-BANDWIDTH-    │
   │   limited by TFLOPS    │      │   BOUND, limited by    │
   │                        │      │   GB/s of HBM          │
   └────────────────────────┘      └────────────────────────┘
        determines TTFT               determines TPOT / ITL
   (time to first token)           (the pace of the stream)

Here’s the arithmetic that makes decode painful. Generating one token requires roughly 2 floating-point operations per model parameter — and requires reading every parameter from GPU memory. In FP16, that’s 2 bytes per parameter: about 1 FLOP per byte moved. An H100 GPU delivers ~989 TFLOPS of dense BF16 compute against 3.35 TB/s of HBM (high-bandwidth memory — the GPU’s main memory) bandwidth — a ratio of ~295 FLOPs per byte. Feed it a workload that offers 1 FLOP per byte and the compute units sit idle 99.7% of the time, waiting on memory.

This gives you the most useful mental model in inference engineering — the decode speed ceiling:

max tokens/sec (single user) ≈ memory bandwidth / model size in bytes

70B model, FP16:  3,350 GB/s ÷ 140 GB  ≈ 24 tokens/sec
70B model, INT4:  3,350 GB/s ÷ ~35 GB  ≈ 96 tokens/sec

No kernel wizardry beats this ceiling for one user on one GPU. You can only raise it three ways: move fewer bytes (quantization), share each byte across more users (batching), or get more than one token per weight-read (speculative decoding). Those three families, plus caching what you’ve already computed (the KV cache), cover nearly every optimization in this article — the rest is implementation detail.

Four terms you need, because every dashboard and paper uses them:

TTFT (time to first token): queue wait + prefill. The “is it frozen?” number.
TPOT / ITL (time per output token / inter-token latency): the decode interval. End-to-end latency ≈ TTFT + TPOT × (output tokens − 1).
Throughput: total tokens/sec across all requests — the cost metric.
Goodput (from the DistServe paper): requests/sec that meet both your TTFT and TPOT targets — the capacity metric. A server can post great throughput while violating SLOs (service-level objectives — your latency targets) on most requests; goodput is what you can actually sell.

One human factor calibrates all of this: people read at roughly 200–300 words per minute — about 5–7 tokens/sec. A chat stream above ~20 tokens/sec already outruns any reader. That slack is a gift to the operator: you can trade per-user speed for batch size (i.e., cost) until you approach the reading floor. The exception is machine-consumed output — code generation and AI agents chaining calls benefit from every token per second you can find.

The KV cache: the memory problem that runs the show

During decode, each new token must attend over the keys and values of every previous token (this is the attention mechanism doing its job). Recomputing those keys and values every step would turn linear generation into quadratic recompute — so every engine caches them. That store is the KV cache, and it is the central resource of LLM serving.

Its size per token is fixed by architecture:

KV bytes per token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_value
                     (2 = one K and one V)

Llama-3-70B (80 layers, 8 KV heads, head_dim 128, FP16):
  2 × 80 × 8 × 128 × 2 bytes = 320 KB per token

→ an 8K-token conversation:    ~2.5 GB   (one request!)
→ a 128K-token context:        ~40 GB    (more than most GPUs)

Now the serving picture snaps into focus: weights are a fixed cost, but KV cache scales with concurrent tokens — batch size × context length. On an 80 GB GPU holding a quantized 70B model, KV cache is what decides whether you serve 10 users or 100. Three families of attack:

1. Stop wasting it — PagedAttention. Early engines allocated each request’s KV cache as one contiguous tensor sized for the maximum possible length. vLLM’s PagedAttention (SOSP 2023) borrowed the oldest idea in operating systems: virtual memory. KV cache lives in fixed-size blocks (16 tokens each); a per-request block table maps logical to physical blocks; blocks are allocated on demand and shared copy-on-write between sequences. Fragmentation waste fell from 60–80% to under 4%, which translated directly into 2–4x throughput over prior systems at the same latency — because the freed memory becomes batch size.

2. Reuse it — prefix caching. Two requests sharing a prefix (the same system prompt, the same few-shot examples, the same conversation history) produce identical KV blocks for that prefix. Engines now hash and reuse them: vLLM’s automatic prefix caching keys blocks by content, and SGLang’s RadixAttention organizes the whole cache as a radix tree over token sequences, reporting up to 6.4x throughput on prefix-heavy workloads like multi-turn chat and RAG. This is also exactly what API providers sell as “prompt caching” — DeepSeek’s price sheet makes the mechanics visible, charging up to ~50–120x less for cache-hit input tokens than cache misses.

3. Shrink it by design — GQA and MLA. Architects attacked the constant factor. Multi-query attention (one shared KV head) proved too lossy; grouped-query attention (GQA) became the sweet spot — Llama-3-70B uses 8 KV heads instead of 64, an 8x cache reduction, explicitly (per Meta’s paper) “to improve inference speed and reduce the size of key-value caches.” DeepSeek’s multi-head latent attention (MLA) compresses KV into a low-rank latent vector for further savings. When you see a model card mention GQA or MLA, that’s inference cost engineering baked into the architecture.

There’s a fourth, blunt option: quantize the cache itself — FP8 KV is nearly free accuracy-wise and halves the 320 KB/token figure. More on quantization below.

Batching: the single biggest throughput lever

Recall the roofline: at batch 1, decode reads 140 GB of weights to produce one token. Serve 64 requests in one batch and the same weight-read produces 64 tokens — decode step time barely moves while memory-bound, so throughput scales almost linearly. Batching is how GPUs earn their keep.

The evolution of how to batch is a lesson in finding the right granularity:

Static batching (the 2021 default): collect N requests, run the batch until all finish. Two failure modes — finished sequences occupy dead slots until the longest request completes, and new arrivals wait for the whole batch to drain. Think of an elevator that won’t open its doors until every passenger has reached their floor.

Continuous batching (Orca, OSDI 2022 — also called in-flight or iteration-level batching): make the scheduling decision every token step. After each step, finished sequences exit immediately and waiting requests join the very next iteration. Orca’s companion trick, selective batching, makes this implementable — token-wise ops (matmuls, layernorms) run on one flattened tensor across all requests while attention runs per-request. Result: 36.9x throughput over request-level batching on GPT-3 175B at equivalent latency. Every serious engine implements this today.

For calibration, measured numbers rather than vendor claims: Anyscale’s often-cited benchmark ladder on OPT-13B showed 4x from optimized kernels alone, 8x from continuous batching, and 23x from continuous batching plus paged KV — on a deliberately high-variance workload versus naive static batching. The 23x is real but is a best-case composite, not a typical production delta.

Chunked prefill (Sarathi-Serve, OSDI 2024) fixed the remaining ugliness: when a 20,000-token prompt arrives, its prefill would monopolize an entire iteration and stall every in-flight stream — users see their tokens freeze. The fix: split prefill into chunks and co-schedule each chunk with the pending decodes, filling a per-iteration token budget with decodes first. Decode steps are memory-bound with idle compute, so piggybacked prefill chunks ride nearly free. This is default behavior in vLLM’s V1 engine, tunable via one knob — a smaller per-step token budget favors smooth streams (ITL), a larger one favors fast first tokens (TTFT).

Disaggregation (DistServe, Splitwise, Mooncake, 2024–25) is the end state at scale: run prefill and decode on separate GPU pools and ship the KV cache between them over NVLink/RDMA. Each pool gets hardware and parallelism tuned to its phase — you can even use compute-heavy GPUs for prefill and cheaper, bandwidth-rich ones for decode (Splitwise measured 2.35x throughput per dollar-and-power budget). DistServe showed up to 7.4x more requests within the same SLOs; Moonshot’s Mooncake runs Kimi this way in production. In 2026 this is a checkbox feature in NVIDIA Dynamo, llm-d, and SGLang — but it only pays past several nodes; single-box deployments should stay colocated with chunked prefill.

        COLOCATED (small scale)            DISAGGREGATED (fleet scale)
   ┌──────────────────────────┐     ┌──────────┐    KV cache    ┌──────────┐
   │  GPU: prefill + decode   │     │ PREFILL  │ ─────────────► │  DECODE  │
   │  mixed via chunked       │     │ pool     │  (RDMA or      │  pool    │
   │  prefill token budget    │     │ compute- │   NVLink)      │bandwidth-│
   └──────────────────────────┘     │ optimized│                │ optimized│
                                    └──────────┘                └──────────┘
                                     scales TTFT                 scales ITL
                                     independently               independently

Quantization: move fewer bytes

If decode speed ≈ bandwidth ÷ bytes, then halving the bytes doubles the ceiling. Quantization stores the model’s parameters (and optionally activations and KV cache) in fewer bits: FP16/BF16 (2 bytes) → FP8/INT8 (1 byte) → INT4-class formats (~0.5 bytes). Think of it as shrinking the truck, not upgrading the engine — the road (memory bus) is the bottleneck, so a smaller truck makes more trips per second.

The intuition for why it works at all: a weight is stored as a low-bit integer plus a shared scale factor per small group (group size 128 is the de facto standard — per-group scales stop one outlier from wrecking the whole tensor’s precision). The interesting engineering is in what’s hard:

Weights are easy. Roughly normal distributions, few outliers; 8-bit weight quantization is essentially free, and careful 4-bit costs little.
Activations are hard. At model scales above ~7B parameters, transformers develop systematic outlier channels carrying values up to ~20x larger than the rest. Naive INT8 activations collapse accuracy. Every famous method is a workaround: LLM.int8() splits outlier columns into FP16; SmoothQuant migrates the difficulty into weights with an offline rescaling; FP8’s exponential value spacing absorbs moderate outliers natively (one reason the industry standardized on FP8 for Hopper-class serving).
The two 4-bit classics differ in philosophy. GPTQ quantizes layer by layer using second-order (Hessian) information from calibration data, compensating each rounding error in the remaining weights. AWQ observes that ~1% of weight channels matter disproportionately — identified by activation magnitudes — and protects them with an equivalent rescaling instead of mixed precision. In practice both land within noise at 4-bit; kernel support decides.

What accuracy actually costs, from published evaluations: 8-bit is under 1% degradation; good 4-bit typically retains ~95–99% depending on model and method, losing ground first on math, code, and long-context reasoning. Two honest caveats. First, heavily-trained recent models (15T-token Llama-3 class) degrade more under low-bit quantization than older ones — less redundancy to absorb rounding. Second, matched benchmark averages can hide per-sample “flips” where the quantized model disagrees with the original; if your use case is sensitive, gate on KL-divergence or task-level evals, not MMLU deltas.

One hardware truth that surprises people: your “4-bit” model usually isn’t multiplied in 4-bit at all. Hopper dropped the INT4 tensor cores Ampere had — and even where they exist, weight-only quantization can’t use them, because activations stay in FP16. The kernel stores weights in 4 bits and dequantizes them to FP16 in registers to multiply; the win is purely bandwidth. That’s also why 4-bit weight-only quantization shines at small batch (bandwidth-bound) and fades at large batch or prefill (compute-bound), where W8A8 (8-bit weights and activations) formats like FP8 — which double effective tensor-core FLOPS — take over. Blackwell’s native FP4 tensor cores are the step change: 4-bit arithmetic acceleration, not just 4-bit storage, and models now ship natively in block-scaled 4-bit formats (OpenAI’s gpt-oss-120b ships MXFP4 weights that fit a single 80 GB GPU).

For local inference, the GGUF ecosystem (llama.cpp, Ollama) uses block formats like Q4_K_M — the community-default 4-bit, which added only ~0.05 perplexity versus ~0.25 for the older Q4_0 format in llama.cpp’s original k-quants benchmarks. Same principle, tuned for CPUs and consumer GPUs.

Speculative decoding: more than one token per weight-read

The third way around the bandwidth wall exploits the compute that low-batch decode leaves idle — spend it on a gamble:

A cheap draft (a small model, extra prediction heads, or even n-gram lookup) proposes the next γ tokens.
The big model verifies all γ+1 positions in one parallel forward pass — which costs nearly the same wall time as generating one token, because the pass is bandwidth-bound either way.
Accepted tokens are kept; the first rejection truncates the rest, and a corrective token is sampled from the big model’s own distribution.

A common misconception is that speculation approximates the model. It doesn’t: with the standard rejection rule, it is provably lossless — the output distribution is mathematically identical to sampling from the target model alone (Leviathan et al., ICML 2023; DeepMind independently, 2023). It’s branch prediction from CPU design applied to language: mispredictions are discarded, correctness is preserved, and it only pays because the pipeline would otherwise stall. (Some schemes, like Medusa’s “typical acceptance,” deliberately relax exactness for extra speed — know which mode your engine runs.)

Expected tokens per verify step is (1 − α^(γ+1)) / (1 − α) where α is the acceptance rate — at α = 0.8 with γ = 5 drafts, you bank ~3.7 tokens per weight-read. What moves α: predictability. Code and structured text accept far more than creative prose; RAG and summarization are so copy-heavy that even model-free n-gram lookup against the prompt yields ~2.4x speedups for free.

The state of the art moved from separate draft models (an ops burden: vocabulary matching, retraining on every target update) to built-in drafting: EAGLE-3 trains a small module fed by the target’s own hidden states (3–6.5x at batch 1), and frontier models increasingly ship with multi-token-prediction heads trained in — DeepSeek-V3’s MTP head hits 85–90% acceptance for ~1.8x, with zero extra ops.

The trade-off to respect: speculation converts spare bandwidth headroom into latency wins. As batch size grows and the GPU becomes compute-bound, verification turns into pure overhead — Meta’s 2025 production paper cites measurements of speedups collapsing from 1.3x to 0.7x (a slowdown!) as batch grew in earlier systems, and reports a realistic 1.4–2x under real fleet load, versus the 3x+ of batch-1 benchmarks. Interactive endpoints: yes. Saturated batch servers: measure first.

Architecture: inside a modern inference engine

Put the pieces together and here’s what actually runs when you pip install vllm and point traffic at it. Every modern engine (vLLM, SGLang, TensorRT-LLM) converges on the same anatomy:

 HTTP request (OpenAI format)
      │
      ▼
 ┌─────────────┐   tokenize, validate, enqueue;
 │ API SERVER  │◄── token stream back to the client (SSE)
 └──────┬──────┘
        ▼
 ┌─────────────┐   every token step: fit all pending decodes
 │ SCHEDULER   │   first, then prefill chunks; admit, evict,
 └──────┬──────┘   preempt (continuous batching + chunked prefill)
        ▼
 ┌─────────────┐   allocate paged KV blocks on demand,
 │ KV CACHE    │   check the prefix cache, share blocks
 │ MANAGER     │   copy-on-write, evict LRU
 └──────┬──────┘
        ▼
 ┌─────────────┐   fused FlashAttention-class kernels,
 │ MODEL       │   CUDA graphs, tensor-parallel shards
 │ RUNNER      │
 └──────┬──────┘
        ▼
 ┌─────────────┐   temperature / top-p / penalties →
 │ SAMPLER     │   one token per running sequence →
 └─────────────┘   loop back to SCHEDULER

The request lifecycle, step by step:

Admission. The API server (OpenAI-compatible HTTP is the de facto standard) tokenizes the prompt and enqueues the request.
Prefix match. The KV manager checks whether cached blocks already cover a prefix of this prompt (same system prompt as the last thousand requests? — skip that prefill entirely).
Scheduling loop — the heart of the engine, one iteration per token step: fit all in-flight decodes into the step’s token budget first, then pack in prefill chunks from waiting requests; allocate KV blocks on demand; if blocks run out, preempt the newest request (drop its blocks, recompute later) rather than crash.
Execution. The model runner launches the fused kernels — FlashAttention-class attention that tiles through on-chip SRAM instead of round-tripping the score matrix through HBM (exact math, just IO-aware), with CUDA graphs eliminating per-kernel launch overhead in the decode fast path (hundreds of tiny kernels per step make CPU launch costs material at small batch).
Sampling. Logits → temperature, top-p, penalties → one token per running sequence.
Stream & loop. New tokens stream back over SSE (server-sent events); finished sequences release their KV blocks instantly (they become reusable cache); the scheduler admits newcomers; repeat — at thousands of iterations per second.

For models too big for one GPU, the runner shards work: tensor parallelism splits every layer across GPUs within a node (needs NVLink-class bandwidth — two all-reduces per layer sit on the critical path), while pipeline parallelism assigns whole layer ranges to different nodes (cheap traffic, but adds per-token latency). The standard recipe: TP inside a node, PP across nodes. Mixture-of-experts models add expert parallelism — shard the expert bank, route tokens over all-to-all — which is how a 671B-total/37B-active model like DeepSeek-V3 serves affordably: SGLang’s public reproduction on 96 H100s hit 52,300 input and 22,300 output tokens/sec per node, at a computed cost near $0.20 per million output tokens.

Above single engines sits the 2026 orchestration layer: KV-cache-aware routers (NVIDIA Dynamo, llm-d, Ray Serve’s prefix-affinity routing) that send each request to the replica already holding its prefix — because engine-level caches are per-worker, and naive round-robin load balancing silently destroys hit rates.

Production: what it takes to run this for real

Everything above is mechanism. Production is policy — and the difference between a demo and a service shows up in six places.

Choosing a stack. As of mid-2026, the honest decision tree is short. vLLM is the general-purpose default (broadest hardware and model support, the ecosystem’s center of gravity). SGLang wins prefix-heavy traffic — agents, RAG, multi-turn chat with fat system prompts — and DeepSeek-style MoE fleets. TensorRT-LLM buys 10–30% more on all-NVIDIA fleets at the price of lock-in and complexity. llama.cpp/Ollama own the laptop and the air-gapped box. And a migration note: Hugging Face TGI was archived in March 2026 — its README now points at vLLM and SGLang; don’t start anything new on it.

Deployment and scaling. The Kubernetes stack consolidated around vLLM workers behind an inference-aware gateway (the Gateway API Inference Extension routes on live engine metrics — queue depth, KV utilization, prefix overlap — rather than round-robin; llm-d and KServe build on it). The autoscaling rule that saves teams weeks of confusion: CPU and GPU-memory metrics are useless here. CPU idles regardless of load; GPU memory is flat because vLLM pre-allocates ~90% of VRAM for KV cache at startup. Scale on vllm:num_requests_waiting (queue depth) and vllm:kv_cache_usage_perc — the latter crossing ~75–80% predicts latency collapse (preemption storms) before users feel it. And set thresholds to lead demand: a new replica pays minutes of image pull plus weight loading before it serves a single token.

Observability. The golden-signal dashboard for LLM serving: p50/p95/p99 TTFT, p99 ITL, queue depth, KV utilization, prefix-cache hit rate, preemption count, and tokens in/out per second (vLLM exports all of these via Prometheus at /metrics). Percentiles, not means — LLM latency is heavy-tailed, and agent pipelines that chain five model calls multiply tail exposure (the classic “Tail at Scale” math: 1% slow calls × 100-way fan-out = 63% of users hitting the tail). Watch the prefix-cache hit rate especially: a deploy that reorders prompt structure shows up as a silent cost regression there before it shows up anywhere else.

Logging and privacy. Token counts, latencies, and finish reasons carry no user content — log them freely. Full prompt/completion capture is different: it pulls your telemetry stack into GDPR/SOC2 scope. The OpenTelemetry GenAI conventions (still pre-stable, in active development as of mid-2026) get this right by defaulting to no content capture with explicit opt-in. The production pattern: always-on metadata, sampled content capture only for errors and SLO violations, PII redaction before emission, separate short-retention store.

Security. Three serving-layer specifics beyond standard API hygiene. Rate-limit in tokens, not just requests — request cost varies ~1000x with context length, and OWASP’s LLM Top 10 names “unbounded consumption” (denial-of-wallet) as a first-class attack. Scope prefix caches per tenant: a 2025 audit (ICML) found cross-user cache sharing at 7 of 17 API providers, leaking information through response-timing side channels — a fast TTFT on a guessed prefix tells an attacker someone else recently sent it. And in continuous batching, one tenant’s monster prompts inflate every tenant’s ITL — noisy-neighbor control means per-tenant admission limits and priority scheduling, not just auth.

Cost. Ranked by effort-to-savings, the levers compound multiplicatively:

Lever	Typical saving	Catch
Prompt caching	75–90% off repeated prefix input	Only exact-match prefixes; structure prompts static-first
Batch APIs	Flat 50% (OpenAI, Anthropic, Gemini alike)	24-hour async window — for evals, backfills, pipelines
Model right-sizing / routing	10–25x per routed request	Needs eval infrastructure to trust it
Quantization (self-hosted)	~2x capacity per precision halving	Gate with evals on your workload
Spot GPUs	60–91% off compute	30s–2min reclaim notice; batch workloads only
Reserved capacity	~30–60% off baseline	Commit only the traffic floor

A workload that caches its system prompt, routes easy traffic to a small model, and batches its offline jobs can sit at a few percent of naive list-price cost. That, more than any kernel, is why “inference engineer” became a job title.

Code: the ideas made concrete

Three short, production-shaped examples. First, the capacity math you should run before renting GPUs — this is the KV-cache formula and decode ceiling as ten lines of Python:

def plan_capacity(params_b, n_layers, n_kv_heads, head_dim,
                  gpu_mem_gb=80, bw_tbs=3.35, kv_bytes=2, w_bytes=2):
    weights_gb = params_b * w_bytes                      # 70B × 2B = 140 GB
    kv_per_token_kb = 2 * n_layers * n_kv_heads * head_dim * kv_bytes / 1024
    free_gb = gpu_mem_gb * 0.9 - weights_gb              # vLLM reserves ~90%
    kv_budget_tokens = max(free_gb, 0) * 1e9 / (kv_per_token_kb * 1024)
    ceiling_toks = bw_tbs * 1000 / weights_gb            # bandwidth ÷ bytes
    return kv_per_token_kb, kv_budget_tokens, ceiling_toks

# Llama-3-70B on one H100: KV budget clamps to 0 → you need TP or quantization
print(plan_capacity(70, 80, 8, 128))

Line by line: weights cost params × bytes (140 GB for 70B FP16 — already over one 80 GB GPU, which is why the function returns zero KV budget and why 70B-class serving starts at tensor parallelism or FP8). KV per token is the formula from earlier (320 KB here). The last line is the roofline ceiling: ~24 tokens/sec single-stream. Halve w_bytes and watch both capacity and ceiling double — quantization’s value, quantified before you spend a dollar.

Second, serving with vLLM — the flags are where the concepts live:

# 4-way tensor parallel · FP8 KV cache · 8K per-step token budget
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90

(Bash note: comments can’t follow the \ line continuations, so they live above the command.) Every flag maps to a section of this article: --tensor-parallel-size shards layers across 4 NVLinked GPUs; --kv-cache-dtype fp8 halves the KV cache for roughly double the batch capacity; --max-num-batched-tokens is the chunked-prefill token budget (drop it to ~2048 if streams stutter, raise it if TTFT lags); --gpu-memory-utilization is the fraction of VRAM pre-allocated for weights plus KV. Continuous batching and prefix caching aren’t flags — both are default behavior in the V1 engine.

Third, measure what users feel — TTFT and ITL from the client side, using the standard OpenAI-compatible streaming API:

import time, openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="-")
t0, first, prev, gaps = time.perf_counter(), None, None, []

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Explain KV caching in 3 lines."}],
    stream=True,
)
for chunk in stream:
    if not chunk.choices or not chunk.choices[0].delta.content:
        continue                         # skip role-only / finish-reason chunks
    now = time.perf_counter()
    if first is None:
        first = now - t0                 # TTFT: queue + prefill
    elif prev is not None:
        gaps.append(now - prev)          # per-token gaps: the ITL distribution
    prev = now

if first is None or not gaps:
    raise SystemExit("stream too short to measure")
gaps.sort()
print(f"TTFT {first*1000:.0f} ms | p50 ITL {gaps[len(gaps)//2]*1000:.1f} ms"
      f" | p99 ITL {gaps[int(len(gaps)*0.99)]*1000:.1f} ms")

Note that it records the distribution of gaps, not the average — a smooth mean can hide two-second stalls (a preemption, a long prefill cutting in) that the p99 exposes instantly.

The mistakes that account for most real-world pain: benchmarking at batch 1 and extrapolating to production (the regimes are different physics); autoscaling on CPU; putting dynamic content (timestamps, user IDs) at the top of prompts, zeroing every cache; quoting throughput when the SLO conversation is about goodput; enabling speculation on a saturated batch server; and skipping evals after quantizing (“the benchmark looked fine” is how per-answer flips reach users).

How the big players run inference

The same playbook, implemented at wildly different scales — and publicly documented enough to learn from:

Google is the vertical-integration case: seventh-generation TPUs (“Ironwood,” GA late 2025) built inference-first — 4.6 petaFLOPS FP8 and 192 GB of HBM at 7.37 TB/s per chip, scaling to 9,216-chip pods — serving Gemini end to end. Its 33x per-prompt energy reduction in one year decomposes (per Google’s own paper) into ~23x from model and software work — quantization, distillation, better batching, MoE — and only ~1.4x from utilization. Software beat hardware 16-to-1.

OpenAI runs on Azure at Stargate scale (a ~$500B, 10 GW buildout with Oracle and SoftBank) and is co-designing inference-specific silicon with Broadcom — the first chip, revealed June 2026, targets ChatGPT and API serving, not training. Its pricing telegraphs the serving stack: 90% discounts on cached input and 50% on batch traffic exist because prefix caching and off-peak batching are nearly free to provide.

Anthropic is the multi-silicon case study: Claude serves across AWS Trainium (the ~500K-chip Project Rainier cluster, part of over a million Trainium2 chips in total), Google TPUs (a contracted up-to-1M-chip deal), and NVIDIA GPUs simultaneously — three kernel stacks kept numerically consistent, in exchange for capacity resilience and pricing leverage.

Microsoft built Maia 100 with deliberately older HBM2E memory — a cost-optimized chip for internal Copilot-class inference, not a flagship-parity play — and contributed chunked prefill’s ancestor (DeepSpeed-FastGen’s Dynamic SplitFuse) to the open ecosystem.

Amazon turned inference economics into retail: Bedrock sells prompt caching (up to 90% off reads), Intelligent Prompt Routing (easy prompts to cheap models — the cascade pattern as a managed service), and latency-optimized serving on its own Trainium2 silicon (96 GB HBM at 2.9 TB/s per chip).

Meta splits the difference: its MTIA silicon serves ads ranking — its public Llama API’s fast tier runs on specialist hardware from Cerebras (Llama 4 Scout at ~2,650 tokens/sec) and Groq. Those two startups attack the bandwidth wall by deleting HBM entirely: Groq’s LPU keeps everything in on-die SRAM at ~80 TB/s (24x an H100’s bandwidth; independently verified 276 tok/s on Llama 3.3 70B), and Cerebras’ wafer-scale chip holds 44 GB of SRAM at 21 PB/s (969 tok/s on Llama 3.1 405B, verified). The catch is capacity: a 70B model spans hundreds of Groq chips, so the economics demand sustained utilization. NVIDIA’s ~$20B licensing-and-hiring deal for Groq’s technology in December 2025 suggests the incumbent decided the SRAM approach was worth owning rather than ignoring.

The startup layer shows where the differentiation now lives: Together’s ATLAS retrains its speculative drafts on live traffic (acceptance rates drift as workloads do); Fireworks sells custom attention kernels including long-context and AMD variants; Baseten orchestrates capacity across 20+ clouds; Modal snapshots GPU memory to cut serverless cold starts from ~70 to ~12 seconds. None of them train frontier models. All of them sell inference.

Advantages: what optimization actually buys

Worth stating plainly, because each maps to a business outcome: an order of magnitude more throughput per GPU (batching + paging) means an order of magnitude fewer GPUs for the same traffic. Sub-second TTFT and smooth streams are product quality — users abandon frozen chatbots. Smaller quantized models unlock deployment surfaces that didn’t exist (laptops, phones, browsers, air-gapped environments). Cheap tokens change what’s feasible — agent loops that burn 100K tokens per task only make sense below a certain price. And the energy stakes are large: at billions of prompts per day, Google’s 33x per-prompt reduction is the difference between AI as a manageable electrical load and AI as a grid problem.

Limitations: what it can’t do

The ceilings are real. Single-user decode speed is bounded by bandwidth ÷ bytes — past ~4-bit, quality falls off a cliff, so dense-model latency has a floor that only exotic hardware (SRAM ASICs) or better drafts can push. Every technique trades something: batching trades individual latency for aggregate throughput; quantization trades measured-on-your-workload accuracy; speculation trades compute and can go negative under load; disaggregation trades operational complexity and network dependency. Optimization also can’t fix the model — a hallucinating 8B model served at 500 tokens/sec hallucinates faster. And the discipline’s numbers rot: vendor benchmarks (“Nx faster than vLLM”) are usually measured against a competitor’s year-old version. Treat every multiplier in this article as a snapshot with a citation, not a constant of nature.

The future: where inference is heading

Three trajectories look durable over the next 3–5 years. Hardware specializes: every hyperscaler now designs inference-specific silicon (Ironwood, Trainium, Maia, OpenAI/Broadcom), Blackwell-class GPUs make 4-bit a compute format rather than just storage, and the SRAM-based speed tier goes mainstream via NVIDIA’s Groq deal. Models co-design for serving: GQA/MLA (KV shrinkage), MoE (compute decoupling), native FP8/FP4 checkpoints, and built-in speculative heads are all inference engineering moved into the architecture — expect “serving cost” to be a first-class model-card metric. The orchestration layer standardizes: OpenAI-compatible HTTP on top, KV-aware routing and prefill/decode disaggregation underneath (Dynamo, llm-d, Gateway API Inference Extension), the same way web serving standardized around load balancers and reverse proxies two decades ago.

For developers, the skills that compound: GPU roofline reasoning (know which regime — compute, bandwidth, or overhead-bound — you’re in before optimizing), the serving-metric vocabulary (TTFT/TPOT/goodput and how to SLO them), one engine deeply (vLLM is the safest bet), quantization evaluation discipline, and cost modeling. The engineer who can say “this feature costs $0.0004 per request now, and here are the two levers to halve it” is rare and, for the foreseeable future, extremely employable.

FAQ

Why is LLM inference so slow? Generating each token requires streaming every weight from GPU memory: 140 GB for a 70B FP16 model against 3.35 TB/s of H100 bandwidth ≈ 24 tokens/sec, single-stream, regardless of compute. Decode is bandwidth-bound; batching, quantization, and speculation are the three ways around it.

What is the KV cache and why does it matter? It stores every previous token’s attention keys/values so they aren’t recomputed each step. At ~320 KB per token for Llama-3-70B, it’s the resource that caps concurrency — 128K of context costs ~40 GB.

What is the difference between prefill and decode? Prefill processes the whole prompt in parallel (compute-bound, sets TTFT); decode emits tokens one at a time (bandwidth-bound, sets TPOT). They’re different enough that big fleets run them on separate GPU pools.

What do TTFT and TPOT (or ITL) mean? Time to first token (queue + prefill) and time per output token (the stream’s pace). End-to-end ≈ TTFT + TPOT × (tokens − 1). SLO on p99s, not means.

What is continuous batching? Scheduling per token step instead of per request — finished sequences exit and newcomers join every iteration. Orca’s version of this delivered 36.9x over request-level batching; it’s universal now.

What is PagedAttention? Virtual-memory-style KV management in fixed-size blocks, cutting allocation waste from 60–80% to under 4% and converting the reclaimed memory into batch size.

Does quantization make models dumber? 8-bit: under 1% loss. Good 4-bit: ~95–99% retained depending on model and method, weakest on math/code/long-context. Below 4 bits it gets real. Evaluate on your workload — aggregate benchmarks hide per-answer flips.

Is speculative decoding lossless? With rejection-sampling verification, yes — provably identical output distribution. Relaxed modes (Medusa’s typical acceptance) trade exactness for speed; know your engine’s setting.

Should I use vLLM, SGLang, or TensorRT-LLM? vLLM as the default; SGLang for prefix-heavy and MoE workloads; TensorRT-LLM for squeezing all-NVIDIA fleets; llama.cpp/Ollama locally. TGI is archived — migrate off.

How much GPU memory do I need to serve a 70B model? ~140 GB FP16 / ~70 GB FP8 / ~35–40 GB 4-bit for weights, plus ~320 KB per concurrent token of KV cache. Production 70B means multiple GPUs or aggressive quantization.

What is prefill/decode disaggregation? Separate GPU pools per phase, KV shipped between them. Up to 7.4x goodput at fleet scale (DistServe); overkill on a single node.

How do I autoscale LLM inference on Kubernetes? On queue depth and KV-cache utilization from engine metrics — never CPU (idle by design) or GPU memory (pre-allocated, always full). Lead demand: replicas take minutes to warm.

How do prompt caching discounts work across API providers? OpenAI: automatic, ~90% off cached input on flagship tiers. Anthropic: explicit breakpoints, 10%-of-base reads, 1.25–2x write premium. Gemini: implicit on 2.5+, per-model minimums. Universal rule: only exact-match prefixes hit — static content first.

What is goodput and how is it different from throughput? Requests/sec meeting both TTFT and TPOT SLOs (DistServe’s term). Throughput measures cost; goodput measures sellable capacity.

Can I run fast LLM inference on a laptop or CPU? Yes — 4-bit GGUF models via llama.cpp/Ollama stream faster than reading speed on a laptop for 7–8B models. The limit is concurrency, not speed: move to vLLM/SGLang when real traffic arrives.

How do Mixture-of-Experts models change inference costs? Compute of a small model, memory of a huge one (DeepSeek-V3: 37B active, 671B resident). Wins on throughput at high batch with expert parallelism; loses on footprint and batch-1 latency versus a distilled dense model.

Summary

LLM inference is a memory-bandwidth problem wearing a compute costume. Prefill and decode are different workloads — one bound by FLOPS, one by GB/s — and every technique in the field is one of four moves: waste less memory (PagedAttention, prefix caching, GQA/MLA), share each byte (continuous batching, chunked prefill, disaggregation), move fewer bytes (quantization of weights and KV cache), or earn multiple tokens per read (speculative decoding). Production adds the policy layer: goodput-based SLOs on p99s, engine-native autoscaling signals, tenant-scoped caches, token-denominated rate limits, and cost levers that compound to 10–100x. The companies serving AI at planetary scale — and the startups undercutting them — are all playing variations of this exact playbook.