MLMei Lin
Cutting LLM serving costs 60% with continuous batching
How vLLM's paged attention and dynamic batching transformed our inference economics.
TB
Tom Becker
Author@tom
Networking & service mesh
Packets, proxies, and p99 latency for model gateways.
2
Stories
403
Views
13
Likes
2
Followers
Contributions
MarAprMayJunJulAug
10 contributions in the last 6 monthsLessMore
DODaniel Okoro
Autoscaling inference with KEDA and custom metrics
Scale on queue depth, not CPU. Your wallet will thank you.
DODaniel Okoro
Running stateful inference on Kubernetes without tears
StatefulSets, warm model caches, and graceful draining for GPU pods.
CRCarlos Ruiz
MIG vs time-slicing: partitioning GPUs for inference
When to slice an A100 into seven, and when not to.
AKAisha Khan
The four golden signals for model inference
Latency, traffic, errors, saturation — adapted for GPU serving.
MLMei Lin
Speculative decoding in production: a field report
Draft models, acceptance rates, and the latency wins that survived contact with reality.