When chain-of-thought helps, and when it only adds tokens

Chain-of-thought prompting asks the model to write intermediate steps before the answer. The steps are output tokens. Output tokens are the expensive side of a GPT-4-class bill, and they are also latency. The steps help when the task is a small chain of judgments the model gets wrong if it jumps. They do not help when the task is a lookup, a format, or a summary of a page you already provided. Asking every route to "think step by step" is how a cheap classification becomes a short essay.

The useful case looks like arithmetic or policy, not like style. A refund decision that depends on three facts — delivered date, condition, whether the customer asked for money — is the kind of thing a model fumbles if it emits the decision word first. If you let it write the three facts down, in order, from the ticket, and only then pick a label, the label tracks the facts more often. You can check this. Run twenty tickets where you already know the label. Count how many the direct prompt gets wrong and how many the stepped prompt gets wrong. If the stepped prompt fixes a real slice, keep it on that route. If it fixes none, you are buying paragraphs.

The useless case looks like extraction. "Copy the order number into the field order_id" does not become more accurate because the model narrates its search. It becomes longer. The failure mode of extraction is usually a missing delimiter or a document that contains two numbers, and the fix is a clearer spec: "the order number is the five-digit integer labeled Order, not the phone number." A paragraph of reasoning will still sometimes pick the phone number, and you will have paid for the paragraph. Fix the spec. The JSON note is that kind of fix.

Hide the steps if you do not want customers to read them. Some APIs let you separate a reasoning channel from the answer, or you can ask for a JSON object with a "notes" field and a "reply" field and show only "reply" in the product. If you do that, cap the notes. "At most four short lines" keeps a model from writing a memo. Uncapped chain-of-thought on a chatty model is an output-token leak. At GPT-4.1's published $8 per million output tokens, an extra 300 tokens on 20,000 calls is 300 × 20,000 / 1,000,000 × $8, which is $48 a day. That is a fair price for fewer bad refund decisions and a bad price for narration nobody reads. Do the division for your own volume before you turn it on globally.

Do not ask for steps as a way to seem rigorous. A model can write plausible steps toward a wrong fact. The steps feel like evidence and are not. If the fact has to be true, put the source in the prompt and tell the model to use only that source. Steps that say "I recall that the window is 30 days" are worse than a constraint that says "The window in the policy block is the only window you may state." Chain-of-thought does not install a memory you can audit. A quoted policy does.

There is a cheap version and an expensive version. The cheap version is a private checklist that does not dump a full essay: "Check delivered date, condition, and whether money was requested. Then choose the label." You still get the ordering of the judgment, and the output can be the label alone if your API or your instructions hold the line. The expensive version is "Explain your reasoning in detail, consider alternatives, and then conclude." That sentence is a request for length. Length is sometimes what a human reviewer wants. It should not be the default on a route whose product is a label.

Evaluate with the steps hidden from the scorer, if you are scoring. If you grade the model's explanation for fluency, you will prefer the verbose prompt for a reason that has nothing to do with the decision. Grade the decision against the twenty tickets. Keep a note of how many output tokens each variant used. A variant that ties on accuracy and loses on tokens is not a tie. Ship the shorter one. The temperature note belongs next to this one, because people reach for sampling settings when the real issue is that the prompt asked for an essay or forgot a rule.

Where steps keep paying for themselves, make them a schema instead of a monologue. "facts_used: up to three short strings. decision: refund or other. missing: a string or null." You get the intermediate commitments, you can log them, and you can reject a decision that cites a fact not in the ticket. A free-form paragraph is harder to reject automatically. Structure is doing the work people hope the word "carefully" will do. The counter on the home page will show you the input cost of that schema. The output cost is the part you must sample, because this page does not invent an output length.

Turn chain-of-thought off on routes where you have not seen a before-and-after. Leave it on, with a cap, where the before-and-after moved the error rate enough to notice in twenty cases. Revisit the decision when you change models. A newer model may not need the scaffolding, and you will keep paying for it until someone deletes the sentence. Put the reason in a comment next to the prompt: "Steps kept on 2026-10-08 because the direct prompt missed 5 of 20 mixed refund cases." A future editor can rerun the twenty and remove the tax when it stops earning.

The phrase "think step by step" is not a strategy. It is a switch that spends output tokens to buy intermediate commitments. Buy them when the commitment is checkable and the direct answer is wrong. Do not buy them because a guide once said reasoning prompts are more advanced. Advanced is not a line item. Tokens are.