Inside LLM Reasoning: What Chain-of-Thought Can and Cannot Tell Us

Chain-of-thought is a performance technique, not a brain scan

When a language model is asked to solve a multi-step problem, it may produce an intermediate sequence of calculations, assumptions and deductions before giving an answer. This is commonly called chain-of-thought reasoning. The technique can improve performance because it gives the model more tokens in which to decompose a task. On an arithmetic word problem, for example, writing out quantities and operations can reduce the chance that a crucial step is skipped. On a planning task, listing constraints may help the model avoid proposing an impossible schedule.

The mistake is to treat that visible sequence as a direct transcript of the computation that caused the answer. Large language models generate each token from patterns encoded in their parameters and the context available at that moment. Their explanations are outputs of the same predictive machinery as their answers. A fluent account of why Paris is the capital of France does not expose a discrete internal record labelled ‘capital lookup’; it supplies a plausible textual justification. Chain-of-thought can therefore be useful without being a faithful window into the model.

This distinction matters operationally. If a system reaches the right diagnosis, recommendation or proof but offers an inaccurate explanation, reviewers may approve it for the wrong reasons. Conversely, a model can produce an awkward or incomplete rationale while its answer remains correct. Visible reasoning should be treated as evidence to examine, not telemetry from an artificial mind. It is closer to a witness statement than a surveillance recording: potentially informative, sometimes persuasive and always open to corroboration.

Why extra reasoning tokens often improve results

Complex questions contain dependencies that are difficult to resolve in a single leap. Consider a retailer deciding whether a promotion was profitable. The model must separate revenue from gross profit, account for discounts, include fulfilment costs and compare the result with a baseline. A terse answer may conflate those quantities. An explicit sequence creates checkpoints: units can be aligned, intermediate totals can be inspected and contradictory assumptions can be noticed before the final figure is generated.

The gains are not uniform. Chain-of-thought prompting tends to help most on tasks with compositional structure, such as mathematics, symbolic manipulation, code tracing and constraint satisfaction. It may add little to straightforward factual retrieval, where more generated text creates more opportunities to introduce an error. Asking for ten steps to identify the author of 1984 does not make the answer more reliable; it merely encourages the model to decorate a simple retrieval with unnecessary narrative.

There is also a cost. A response using 1,000 reasoning tokens consumes more time and compute than one using 100, particularly when deployed across millions of requests. Longer traces can wander, lock onto an early mistake or overfit to the wording of a prompt. Providers therefore face a practical trade-off between accuracy, latency and expenditure. The best strategy is often adaptive: use concise generation for routine queries, then allocate more test-time computation to problems whose difficulty or risk justifies it.

Plausible explanations can be post-hoc stories

Models are trained on vast quantities of human text in which answers are accompanied by explanations. They learn the shape of a convincing rationale: define the issue, cite relevant facts, connect them logically and state a conclusion. That skill can mask a lack of causal fidelity. A model may first settle on an answer through pattern recognition, then generate a rationale that makes the answer appear deliberate. The text can be coherent even when the supposed reasoning did not determine the result.

Prompt interventions reveal this weakness. If an irrelevant hint changes the final answer but disappears from the explanation, the visible trace has omitted an influential factor. Suppose a model is told, without evidence, that ‘a trusted expert believes option B is correct’. If it switches from A to B yet later claims that B followed solely from the data, the rationale is not faithful. Similar tests use biased answer formats, misleading demonstrations or altered ordering to see whether hidden cues affect decisions without being acknowledged.

Models may also rationalise mistakes. After selecting an incorrect medical code or legal interpretation, a system can invent supporting details, misstate a rule or perform arithmetic that conveniently lands on the chosen result. Fluency makes these failures dangerous because readers tend to equate specificity with evidence. The remedy is not to demand an even longer explanation. It is to check claims against source documents, execute calculations independently and test whether the conclusion survives controlled changes to the prompt.

Accuracy and faithfulness are separate measurements

Evaluation becomes clearer when two questions are separated. Accuracy asks whether the final answer is correct. Faithfulness asks whether the stated rationale reflects the factors that actually drove that answer. A model can score highly on one and poorly on the other. For a benchmark with 1,000 questions, a system might answer 850 correctly, yet controlled perturbations could show that many accompanying explanations omit decisive prompt cues. Reporting only 85 per cent accuracy would conceal that weakness.

Rationale quality introduces further dimensions. Is each step logically valid? Are cited facts supported? Does the explanation cover the essential steps without irrelevant material? Human reviewers can score these properties, but they bring inconsistency and cost. Automated judges scale better, although they may share biases with the model being assessed. A polished but subtly invalid proof can fool both. Evaluation should therefore combine final-answer metrics, step-level checks and targeted human review rather than collapsing performance into a single number.

The importance of each metric depends on the application. A spelling assistant may need little explanation if its corrections are easily reversible. A credit decision, clinical recommendation or safety-critical engineering calculation demands traceable evidence and accountable procedures. Even there, a model-generated chain-of-thought is not an audit log. A stronger record includes retrieved sources, tool calls, input versions, calculation outputs and policy rules. Those artefacts can be reproduced; free-form reasoning usually cannot.

Behavioural tests offer stronger evidence

The most persuasive evidence of capability comes from what a model can reliably do across controlled variations. Evaluators should alter names, numbers, surface wording and answer order while preserving the underlying problem. If performance collapses when a familiar template is paraphrased, the original result may reflect pattern matching rather than robust abstraction. If the model handles novel values and structures, confidence in the underlying capability rises.

Counterfactual testing is especially revealing. Change one relevant fact and verify that the answer changes appropriately; change an irrelevant fact and verify that it remains stable. In a loan example, increasing verified income may legitimately affect affordability, while changing the applicant’s surname should not. In a physics problem, doubling mass should influence some quantities but not others. These tests probe sensitivity to causal features without requiring access to an internal monologue.

Repeated sampling also exposes uncertainty that a single response hides. A model that gives the same answer in 95 of 100 runs is behaving differently from one that splits 55 to 45, even if the first displayed response is correct in both cases. Self-consistency methods exploit this by generating multiple candidate solutions and selecting the most common answer. Agreement is not proof—many samples can share the same misconception—but variation provides a practical signal for escalation, abstention or additional verification.

Tools, process supervision and adversarial evaluation

External tools convert some reasoning claims into testable operations. Instead of trusting a model’s long division, a system can call a calculator. Rather than accepting a quoted statute, it can retrieve the current text from an authoritative database and attach the relevant passage. Code can be executed against unit tests; schedules can be checked by a constraint solver. The model still chooses how to use these tools, but the resulting evidence is more concrete than prose alone.

Process supervision offers another route. During training or evaluation, reviewers assess intermediate steps rather than only the final answer. A mathematical solution that accidentally reaches the correct number after two cancelling errors should not receive full credit. Step-level feedback can teach models to favour valid decompositions and enables evaluators to locate failure points. The trade-off is substantial annotation effort: judging a ten-step derivation requires more expertise and time than checking one final result.

Adversarial testing complements both approaches. Red teams can introduce misleading premises, conflicting documents, prompt injections and superficially authoritative claims. They can test whether a model cites a source that does not support its answer or follows instructions hidden inside retrieved content. Sandboxed environments are particularly valuable for agents: evaluators can observe whether a system completes a task, respects permissions and recovers from errors. Successful action under constraints is stronger evidence than an eloquent declaration of intent.

Designing systems that do not overtrust narration

Production systems should separate the answer shown to a user from the evidence used to validate it. That may mean presenting a concise recommendation alongside citations, calculations and clearly labelled assumptions rather than exposing a sprawling internal rationale. Users need grounds they can inspect, not an anthropomorphic performance of deliberation. For sensitive tasks, the interface should distinguish verified facts, model inferences and unresolved uncertainty.

Developers should define escalation thresholds before deployment. If repeated samples disagree, a required source is unavailable or a tool result conflicts with the generated answer, the system can abstain or route the case to a person. Monitoring should track not only average accuracy but performance by task type, input distribution and risk category. A model that achieves 90 per cent overall may still fail half of the rare cases that matter most, such as unusual drug interactions or ambiguous contractual clauses.

Chain-of-thought remains valuable as a problem-solving scaffold and diagnostic clue. It can reveal common misconceptions, suggest better prompts and help reviewers identify where a solution appears to break down. What it cannot provide is privileged access to the model’s true internal computation. Sound evaluation rests on reproducible behaviour, verified artefacts, controlled interventions and outcomes measured across many cases. The more consequential the decision, the less weight should be placed on narration alone.