Edge inference: serving models at the CDN
Latency budgets when 50ms of network is unacceptable.
TB
Tom Becker
May 13, 2026·1 min read·88
Some experiences die at 100ms. For those, you push small models to the edge and accept the operational complexity that follows.
What fits at the edge
Embedding models for search.
Small rerankers.
Guardrail classifiers.
Keep the big generative models centralized; ship the latency-critical, small ones out.
TB
Written by
Tom BeckerNetworking & service mesh
Packets, proxies, and p99 latency for model gateways.
2 followers · 2 stories