# Inside vLLM: why inference systems, not models, decide your costs

- Published: 2026-07-18
- Authors: CORTEXA
- Category: Engineering
- HTML: https://researchhub-vert.vercel.app/blog/vllm-internals-inference-throughput

The model determines what is possible. The serving stack determines what it costs. Most teams spend all their attention on the first.

A walkthrough of vLLM's architecture — *Inside vLLM: Anatomy of a High-Throughput LLM Inference System* — resurfaced on Hacker News this week and drew a large audience. It is worth understanding why a systems post about inference draws that kind of attention.

## The thing teams get wrong

Most teams choosing a model compare quality benchmarks and price-per-token. Then they self-host and discover that their actual throughput is a fraction of what the arithmetic suggested, because throughput on real traffic is decided by the serving layer, not the weights.

The gap is not small. The difference between a naive server and a well-tuned one on the same hardware and the same model is routinely several times, not several percent.

## Where the throughput goes

Three ideas do most of the work in modern inference servers.

**Continuous batching.** Naive batching waits for a batch to fill, runs it, and waits again. Requests finish at different times, so the batch is mostly idle slots. Continuous batching evicts finished sequences and admits new ones every step, keeping the batch full.

**Paged KV cache.** The attention cache grows with sequence length, and if you allocate it contiguously you must reserve for the worst case. Paging it — allocating fixed-size blocks on demand, like virtual memory — removes the reservation and dramatically raises how many concurrent sequences fit in the same GPU memory.

**Scheduling under memory pressure.** With paging, admission becomes a real decision: which requests to run now, which to preempt, which to queue. This is the part that most affects tail latency, and the part most often left at defaults.

```mermaid
flowchart TD
  R[Incoming requests] --> S[Scheduler]
  S -->|admit| B[Running batch]
  S -->|queue| Q[Waiting queue]
  B --> K[Paged KV cache]
  K -->|pressure| S
  B -->|finished| O[Stream out]
  Q -->|slot free| B
```

## Why this matters more than model choice for cost

Model choice sets a floor on quality and a rough scale for compute. But the same model on the same GPU can cost you very different amounts depending on:

- how well your batch stays full,
- how much KV memory you waste on reservation,
- whether your long requests starve your short ones,
- whether you are recomputing prefixes you could have cached.

None of those appear on a model card.

## The practical takeaway

If you are self-hosting and your costs are higher than expected, the first place to look is not a smaller model. It is your batch occupancy and your KV cache utilisation. Those two numbers explain most inference bills, and both are measurable in an afternoon.

The original writeup is worth reading in full if you operate inference infrastructure — it is one of the clearer explanations of why these systems are shaped the way they are.