Cutting LLM serving costs 60% with continuous batching

When we first deployed our 13B parameter model, GPU utilization hovered around 30%. Requests arrived one at a time, each waiting for the previous to finish. We were paying for idle silicon.

The problem with static batching

Static batching forces you to choose between latency and throughput. Wait to fill a batch and tail latency suffers; ship small batches and you waste compute. Continuous batching sidesteps the tradeoff entirely.

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-13b",
          gpu_memory_utilization=0.92,
          max_num_seqs=256)
params = SamplingParams(temperature=0.7, max_tokens=512)

What changed

• GPU utilization rose from ~30% to ~85% under production traffic.

• p50 latency dropped because new requests join the running batch immediately.

• We retired 4 of every 10 GPUs — a 60% cost reduction at the same SLO.

The cheapest GPU is the one you didn't have to rent.

In the rest of this post I'll walk through the KServe + vLLM setup, the autoscaling signals we watch, and the two footguns that cost us a weekend.