AKAisha Khan
The four golden signals for model inference
Latency, traffic, errors, saturation — adapted for GPU serving.
Topic
Observability
3 stories
3 stories
Latency, traffic, errors, saturation — adapted for GPU serving.
Scale on queue depth, not CPU. Your wallet will thank you.
Error budgets when your dependency is a probabilistic model.