The simplest form of test-time compute is chain-of-thought: the model generates reasoning steps before the final answer. More sophisticated approaches include: tree-of-thought (exploring multiple reasoning paths and selecting the best), self-consistency (generating multiple answers and voting), and iterative refinement (the model critiques and revises its own output). Each approach uses more tokens (= more compute = more cost) but produces better results.
Models like o1 (OpenAI) and Claude with extended thinking generate internal reasoning tokens that the user doesn't see. These "thinking tokens" let the model decompose complex problems, check its work, consider edge cases, and revise its approach — all before producing the visible response. The cost is higher (you pay for thinking tokens) and latency is longer, but accuracy on math, coding, and reasoning tasks improves dramatically.
Research suggests that test-time compute follows its own scaling laws: doubling inference compute (thinking time) produces predictable improvements in accuracy, analogous to how doubling training compute improves pre-training loss. This means you can choose your quality-cost trade-off per query: simple questions get fast, cheap answers; complex questions get longer, more expensive reasoning. This dynamic allocation is more efficient than using the same compute for every query.