MLMei Lin
Cutting LLM serving costs 60% with continuous batching
How vLLM's paged attention and dynamic batching transformed our inference economics.
Topic
Cost Optimization
3 stories
3 stories
How vLLM's paged attention and dynamic batching transformed our inference economics.
When to slice an A100 into seven, and when not to.
INT8, FP8, and AWQ — measuring what you actually lose.