Skip to main content
Zubnet AI › Learn › Wiki › Continuous Batching
Infrastructure

Continuous Batching

Also known as: Dynamic Batching, In-flight Batching, Iteration-level Scheduling
An LLM serving technique that adds new requests to a running batch and removes finished ones at every generation step, keeping the GPU full.

Why it matters

Continuous batching is the main reason modern inference servers can serve many users from one GPU at reasonable cost. Benchmarks published by Anyscale in 2023 showed up to 23 times more throughput than static batching, along with lower latency. Every serious LLM server now uses it: vLLM, Hugging Face TGI, SGLang, NVIDIA's TensorRT-LLM (which calls it in-flight batching) and the llama.cpp server.

If you call a hosted LLM API, your request is almost certainly sharing a GPU batch with other people's requests, token by token.

Deep Dive

GPUs are most efficient when they process many sequences at once. For an image classifier, batching is easy: group requests, run them together, return all the results. Text generation is harder, because each request produces a different number of tokens over many steps. One user wants ten tokens, another wants two thousand.

With naive (static) batching, the server waits to collect a batch, then runs it until the longest request finishes. Short requests sit idle in finished slots, and new arrivals wait outside until the whole batch is done. Continuous batching fixes this by making the scheduling decision at every single token step: finished sequences leave immediately, and waiting requests take their place.

How It Works

The idea was introduced as iteration-level scheduling in the Orca paper (OSDI 2022). The server keeps a pool of active sequences. At each step, it:

  1. Runs one forward pass that produces the next token for every active sequence.
  2. Removes sequences that hit an end token or their length limit.
  3. Admits waiting requests into the freed capacity.

Because each sequence has its own KV cache, sequences at completely different stages (one on its 5th token, another on its 900th) can share a forward pass.

Prefill and Decode

A new request must first process its whole prompt (prefill) before it can generate (decode). Prefill is heavy: a 20,000-token prompt joining the batch can stall everyone else's next token. Servers handle this with chunked prefill, which splits long prompts into pieces processed across several steps and mixed with ongoing decodes. Large deployments sometimes put prefill and decode on separate workers.

The Memory Limit

How many sequences fit in a batch is limited less by compute than by KV cache memory, which grows with every token of every active sequence. This is why continuous batching and PagedAttention arrived together in vLLM. Allocating cache in small blocks on demand, instead of reserving the maximum length for each request, lets far more sequences fit. When memory runs out, a server may pause some sequences and resume them later.

Throughput vs. Latency

Bigger batches raise total tokens per second but slow each individual stream somewhat, since every step does more work. Under light load, each user gets fast output. Under heavy load, per-user speed drops and, past a point, requests queue and time to first token climbs. Operators tune limits such as the maximum number of concurrent sequences to stay within their latency targets.

What It Means in Practice

  • If you self-host with vLLM or a similar server, send requests concurrently instead of one at a time. The server batches them for you, and total throughput can be many times higher than with a sequential loop.
  • Benchmark at realistic concurrency. Single-request speed says little about capacity.
  • Set max_tokens on requests. Long generations hold memory in the batch.
  • Don't confuse this with a provider's Batch API, which is about asynchronous jobs at a discount, not GPU scheduling.
← All Terms
ESC