Chain-of-Thought Prompting

What Is Chain-of-Thought Prompting?

Chain-of-thought prompting is the technique of instructing a language model to work through a problem step by step before giving its final answer, rather than answering immediately. Making the intermediate reasoning explicit measurably improves accuracy on math, logic, planning, and multi-step tasks, and it leaves a visible trail you can inspect when the answer is wrong. In the original 2022 paper from Wei et al. at Google, a 540B-parameter model prompted with just eight chain-of-thought exemplars reached state-of-the-art accuracy on the GSM8K math benchmark, beating even a finetuned GPT-3 with a verifier [1].

Key Takeaways

• The gain comes from generation order. Producing reasoning tokens first gives the model computed intermediate results to condition on when it commits to an answer.
• It helps most on multi-step problems: arithmetic, logic, planning, debugging. On simple lookups and classification it adds tokens without adding accuracy.
• The visible trail is a debugging tool. When the answer is wrong, the reasoning usually shows which step broke, which a bare answer never reveals.
• Reasoning models have absorbed the technique into training. They generate internal chains of thought by default, so prompting for it explicitly matters less on those models than it did in the early days.

How It Works

Language models generate one token at a time, each conditioned on everything before it. When a model answers a multi-step question directly, it must land on the conclusion in a handful of tokens, with no room to compute the intermediate steps. Chain-of-thought prompting changes the shape of the output: the model first writes out the sub-steps, and each written step becomes context for the next. By the time the final answer arrives, it is conditioned on a worked solution instead of a hunch. In effect, the model spends more compute on the problem by writing more tokens.

Triggering it is simple. Zero-shot chain-of-thought appends an instruction like "think step by step before answering." That phrase alone carries surprising weight: Kojima et al. found in 2022 that appending "Let's think step by step" lifted InstructGPT's zero-shot accuracy from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K [2]. Few-shot prompting versions include worked examples that demonstrate the reasoning style, which also teaches the model how granular the steps should be. Structured variants ask for reasoning inside dedicated tags or a scratchpad field, then have code strip that section and keep only the answer. Sampling helps too: self-consistency, which generates several chains of thought and takes the majority answer, boosted chain-of-thought performance by 17.9 points on GSM8K, 11.0 on SVAMP, and 12.2 on AQuA in Wang et al.'s 2022 study [3].

The technique proved important enough that it moved from prompting into training. A reasoning model is trained to produce long internal chains of thought before responding, spending variable amounts of compute depending on difficulty. For those models an explicit "think step by step" is mostly redundant, though asking them to show or structure their reasoning still has value for auditability.

Example

A platform team uses a model to triage production incidents. Asked directly "which service caused this outage?" with a bundle of logs, the model tends to name whichever service appears most often in the error messages, which is usually a victim, not the cause. They rewrite the prompt to force a chain: list the observed symptoms in timestamp order, identify which service logged the earliest anomaly, trace which upstream dependency could produce that anomaly, and only then name a root-cause candidate with the supporting evidence. Accuracy on their replayed incident set improves noticeably, and when the model still gets one wrong, the on-call engineer can see exactly which inference step went off the rails and correct it.

What People Get Wrong

The big misconception is reading the chain of thought as a faithful transcript of the model's computation. It is not. Models sometimes produce a plausible-looking rationale and an answer that does not actually follow from it, or reach an answer for reasons the written chain never mentions. Treat the chain as a quality-improving scaffold and a useful debugging signal, never as proof of correctness. Verification still has to come from tests, checks, or review outside the model.

FAQ

Is chain of thought the same as a reasoning model? No. Chain of thought is a prompting technique any model can follow at inference time. A reasoning model has that behavior trained in and allocates its own thinking budget. The technique came first; the models productized it.

Does chain-of-thought prompting still matter in 2026? Yes, with narrower scope. On reasoning models the generic "think step by step" adds little, but domain-specific reasoning structure, like the ordered triage steps in the example above, still steers models toward the right procedure. On fast non-reasoning models used for cost reasons, classic chain of thought remains one of the cheapest accuracy upgrades available.

Should I show the reasoning to end users? Usually not. Chains are verbose, occasionally wrong in embarrassing ways, and not written for readability. The common pattern is to generate reasoning in a structured field, log it for debugging and evals, and surface only the final answer.

Sources

1. Wei et al. (Google), arXiv (Chain-of-Thought Prompting). "540B-parameter model with eight chain-of-thought exemplars achieved state-of-the-art GSM8K accuracy, surpassing finetuned GPT-3 with a verifier." https://arxiv.org/abs/2201.11903. Accessed August 2026.
2. Kojima et al., arXiv (Large Language Models are Zero-Shot Reasoners). "'Let's think step by step' raised InstructGPT zero-shot accuracy from 17.7% to 78.7% on MultiArith and 10.4% to 40.7% on GSM8K." https://arxiv.org/abs/2205.11916. Accessed August 2026.
3. Wang et al. (Google), arXiv (Self-Consistency). "Self-consistency boosted chain-of-thought performance by +17.9% on GSM8K, +11.0% on SVAMP, and +12.2% on AQuA." https://arxiv.org/abs/2203.11171. Accessed August 2026.