Batch API
Why it matters
A lot of LLM work doesn't need an answer in seconds: running an evaluation suite, classifying a million support tickets, generating embeddings for a document archive, producing synthetic training data, summarizing yesterday's transcripts. Paying real-time prices for that work is simply wasteful.
Batch jobs also have separate, much higher rate limits, so they don't use up the quota that your live product depends on. For data pipelines, switching to batch is often the single easiest cost reduction available.
Deep Dive
A batch API lets you submit many requests at once, thousands or more, and collect the results later instead of waiting for each response in real time. You upload the requests, the provider processes them when it has spare capacity, and you download the outputs when the job is done, usually within 24 hours and often much sooner.
In exchange for giving up immediacy, you get a large discount. As of 2026, OpenAI, Anthropic and Google all price batch requests at about 50% off their standard rates.
How It Works
The flow is similar across providers:
- Prepare the requests. Each one is a normal API request plus a
custom_idthat you choose. - Submit the batch. OpenAI takes an uploaded JSONL file with one request per line, while Anthropic's Message Batches API takes a list of requests directly.
- Poll for status, or just wait.
- Download the results and match them to your inputs by
custom_id. Results aren't guaranteed to come back in the original order.
One line of an OpenAI-style batch file looks like this:
{"custom_id": "ticket-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "MODEL_NAME", "messages": [{"role": "user", "content": "Classify this ticket: ..."}]}}
Provider Details
- OpenAI Batch API (April 2024): JSONL files, a 24-hour completion window, and support for chat, embeddings and other endpoints.
- Anthropic Message Batches (October 2024): up to 100,000 requests per batch, with most batches finishing in under an hour.
- Google Gemini offers batch mode through the Gemini API and Vertex AI, and Amazon Bedrock has batch inference for the models it hosts.
If you run your own GPUs, you don't usually need a special batch API: you send requests as fast as the server's continuous batching can handle.
Stacking Discounts
Batch pricing can combine with prompt caching. If every request shares a long prefix, such as the same instructions or the same document, cached tokens are billed at a reduced rate on top of the batch discount. Cache hits in batches are best-effort, because requests are processed in parallel, but putting the shared content first still pays off.
When Not to Use It
Batch is the wrong tool whenever someone, or something, is waiting. That rules out chat, most agent loops (each step depends on the previous one) and anything with a tight deadline. The 24-hour window is a maximum, and although results usually arrive sooner, you can't count on it.
Practical Tips
- Pilot first. Run 50 requests in real time, check the outputs, then submit the full batch. A prompt bug multiplied by a million is expensive.
- Handle partial failure. Individual requests can fail or expire while the rest succeed. Collect the failures and resubmit them.
- Use structured output. With no chance to ask again interactively, you want results you can parse automatically.
- Split huge jobs. Several medium batches are easier to monitor and retry than one enormous one.
- Set
max_tokenssensibly. Runaway outputs are harder to notice when nobody is watching.