The four golden signals for model inference

Google's golden signals translate cleanly to inference if you know where to look.

• Latency: split queue time from compute time — they fail differently.

• Traffic: measure tokens/sec, not just requests/sec.

• Errors: count OOM-kills and timeouts separately.

• Saturation: KV-cache occupancy is your real saturation metric.

Instrumenting with OpenTelemetry

from opentelemetry import metrics
meter = metrics.get_meter("inference")
queue_time = meter.create_histogram("inference.queue_ms")

Once queue time is visible, autoscaling decisions stop being guesswork.