OpenAI-Compatible API
Why it matters
Compatibility removes most of the cost of switching. An app, agent framework or editor plugin written for OpenAI can be pointed at a local model, a cheaper provider or a self-hosted server without rewriting code. That makes it easy to compare models, avoid lock-in and move workloads to wherever price and quality are best.
It's also why local tools work with so much software out of the box. Any app with a "custom OpenAI endpoint" setting can talk to Ollama, LM Studio or vLLM.
Deep Dive
When OpenAI released its Chat Completions API in 2023, its format quickly became the one that most developers and tools were written against: a POST to /v1/chat/completions with a model name and a list of messages, each with a role (system, user or assistant) and content. Other providers and open-source servers found that the easiest way to win adoption was to accept the same format.
The result is a de facto standard. An "OpenAI-compatible" server speaks this protocol even though the model behind it has nothing to do with OpenAI. To switch, you change two settings in your client: the base URL and the API key.
What It Looks Like
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1", # Ollama, or any compatible server
api_key="not-needed-locally",
)
response = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Explain KV caching in one sentence."}],
)
print(response.choices[0].message.content)
The same code works against a different server by changing base_url and model.
Who Supports It
- Local and self-hosted servers: vLLM, Ollama (
localhost:11434/v1), LM Studio (localhost:1234/v1), the llama.cpp server and SGLang. - Inference providers: Together AI, Fireworks AI, Groq, Cerebras, DeepSeek, Mistral, xAI and many others.
- Aggregators and gateways: OpenRouter and LiteLLM expose hundreds of models through one compatible endpoint.
- Other major labs: Google offers an OpenAI-compatible endpoint for Gemini, and Anthropic provides a compatibility layer intended mainly for testing.
Where Compatibility Breaks
"Compatible" usually means the basics work: messages in, text out, streaming. Beyond that, expect differences.
- Tool calling: the format is mostly shared, but reliability, parallel calls and streaming of tool arguments vary by server and model.
- Structured output:
response_formatwith a JSON schema is supported by some servers, partly supported by others and ignored by a few. - Reasoning: there's no agreed field for thinking tokens. Providers return them in different places, or not at all.
- Provider-specific features: prompt caching controls, citations and built-in web search often don't map onto the OpenAI format, so the compatible endpoint exposes less than the provider's native API.
- Parameters and errors: unsupported parameters may be silently dropped, token usage may be reported differently and error codes aren't consistent.
The rule of thumb: for plain chat, trust compatibility. For anything advanced, test the exact feature against the exact server, and use the native API when you need a provider's full capabilities.
A Moving Standard
OpenAI itself has moved on. In 2025 it introduced the Responses API as its recommended interface, with built-in tools and server-side conversation state, while continuing to support Chat Completions. Most of the ecosystem still targets Chat Completions, though some servers have begun to add Responses support. Meanwhile, Anthropic's Messages API has become a second common format, and several providers now offer Anthropic-compatible endpoints as well, largely so that coding agents can run on their models.
So "OpenAI-compatible" is best read as "speaks Chat Completions", a lowest common denominator that isn't going away soon.