Skip to main content
Zubnet AI › Learn › Wiki › AI Gateway
Infrastructure

AI Gateway

Also known as: LLM Gateway, LLM Router, Model Router
A layer between applications and model providers that offers one API for many models and handles routing, fallbacks, cost tracking and access control.

Why it matters

Few serious teams use just one model. They mix providers for cost, quality, speed and resilience. Without a gateway, each application handles its own keys, retries, logging and provider quirks, and nobody can answer simple questions such as "how much did we spend on which model last month?" or "which apps send customer data to which provider?"

A gateway centralizes all of that. Swapping a model becomes a configuration change instead of a code change, an outage at one provider triggers an automatic fallback, and finance and security get a single place to set budgets and policies.

Deep Dive

An AI gateway is a proxy that sits between your code and the LLM providers you use. Your applications send every model request to the gateway, in one format and with one set of credentials. The gateway forwards each request to the right provider, whether that's OpenAI, Anthropic, Google, a cloud platform or your own servers, and returns the response.

It plays the role that an API gateway plays for microservices, adapted to the particular problems of LLMs: many providers with different APIs, per-token billing, rate limits, outages and new models every few weeks.

What a Gateway Does

  • Unified API: one interface, usually OpenAI-compatible, across all providers.
  • Key management: provider keys stay in the gateway. Teams and apps get virtual keys that can be scoped and revoked.
  • Fallbacks and retries: if a provider errors or times out, retry or switch to a backup model.
  • Load balancing: spread traffic across keys, regions or deployments to stay under rate limits.
  • Budgets and rate limits: cap spending and usage per team, user or project.
  • Logging and cost tracking: record every request, token count and cost for observability and billing.
  • Caching: return a stored response for identical requests, and sometimes for semantically similar ones.
  • Guardrails: scan requests and responses for personal data, secrets or policy violations.

Routing

The simplest routing is a fixed rule: this app uses this model, with that one as a backup. More advanced gateways route by cost and difficulty, sending easy requests to a cheap model and hard ones to a strong model. The RouteLLM research project (2024) reported cutting costs by more than 85% on one benchmark while keeping 95% of GPT-4's quality, by learning which queries need the stronger model.

Smart routing has real pitfalls, though. Predicting difficulty from the prompt alone is unreliable. Different models respond differently to the same prompt, so quality becomes harder to evaluate. And switching models in the middle of a conversation throws away the prompt cache, which can cost more than the routing saved.

Products

  • LiteLLM: an open-source proxy and Python SDK, the common self-hosted choice.
  • OpenRouter: a hosted service with a single account and bill for hundreds of models.
  • Portkey, Helicone and Kong AI Gateway: gateways with a focus on observability and governance.
  • Cloudflare AI Gateway and Vercel AI Gateway: gateways built into developer platforms.

Cloud platforms such as Amazon Bedrock, Microsoft Foundry and Google Vertex AI play a similar role within their own catalogs.

Trade-offs

  • Another hop: a little added latency, and one more thing that can go down. A gateway outage takes every model with it.
  • Feature lag: new provider features may not be supported right away, or may be lost in translation to a common format.
  • Data exposure: a hosted gateway sees all your prompts and responses. Check its retention and privacy terms, or self-host.
  • A high-value target: the gateway holds every provider key. Secure and monitor it like any other critical infrastructure.

Do You Need One?

A solo project calling one model doesn't. The need appears with the second provider, the second team or the first surprise bill. A sensible path is to start with a thin wrapper in your own code, then move to a self-hosted or managed gateway once you need shared keys, budgets and fallbacks across several applications.

← All Terms
ESC