Building a model gateway with Envoy and gRPC

A model gateway is the front door to your inference platform. Get it right and everything behind it gets simpler.

Why a dedicated gateway

• Per-tenant rate limiting and quotas.

• Retries with hedging for tail latency.

• Auth, mTLS, and request logging in one place.

rate_limits:
  - actions:
      - request_headers:
          header_name: x-tenant-id
          descriptor_key: tenant

Envoy's gRPC support means you can multiplex thousands of streaming token responses over a handful of connections.