The technical mechanism: during the "prefill" phase of LLM inference, the model processes all input tokens and computes their KV cache entries. Prompt caching stores this KV cache so that subsequent requests with the same prefix skip the prefill for the cached portion. Only new tokens (the user's actual message) need processing. Anthropic, OpenAI, and Google all offer some form of prompt caching.
Most implementations work by detecting matching prefixes automatically or by letting you mark cache breakpoints. The key constraint: only exact prefix matches count. If your system prompt changes by even one token, the cache misses. This means structuring your prompts with the stable parts first (system prompt, documents) and variable parts last (user message) is important for cache hit rates.
Prompt caching delivers the biggest savings when: (1) you have a long, stable prefix (large system prompts, RAG context), (2) you send many requests with that same prefix (chatbots, agents), and (3) input tokens are a significant portion of your costs. For applications with short, unique prompts, caching provides little benefit. For applications that stuff the context window with documents, it's transformative.