Temperature does not fix a vague prompt

Temperature, top_p, and the rest of the sampling settings change which token the model is willing to pick when several tokens are plausible. They do not change what the prompt asked for. A vague prompt at temperature 0 is still vague. It will be vague in the same way every time, which is useful for tests and useless as a cure. If you reach for the sampling slider because the answers feel wrong, write down the wrongness first. Almost always it is a missing constraint, a double format, or a document in the same voice as the assignment.

What temperature actually does, in plain terms: at a low setting the model sticks to the most likely continuation. At a high setting it more often picks a less likely one. For a classification label with one right answer, low is what you want, so retries are comparable and the pilot is not noise. For a brainstorming route where you asked for ten different headlines, a higher setting produces variety you asked for. Using a high setting on the classifier and a low setting on the brainstorm is a common mix-up. Match the setting to whether you wanted a single stable result or a spread.

What it does not do is fill in a spec. If the prompt says "categorize the email" and never lists the categories, temperature 0 will pick some stable wrong taxonomy and stick to it. You will think the model "has a bias." It has a missing list. Write the list. The JSON note shows the shape of that list: the allowed values, the key, the ban on a preamble. After the list exists, set temperature low so the pilot measures the prompt and not the sampler. If you change the prompt and the temperature in the same deploy, you will not know which one moved the error rate.

Invented facts are the case people most want temperature to solve. A model states a delivery date that is not in the record. Turning temperature down makes it state the same invented date more often, or it makes it hedge with equal consistency. The fix is a sentence: do not state a date that is not in the record. Test the ticket where the date is absent. The correct answer is the absence, at any temperature. If the model still invents at temperature 0, the sentence is not in the request it saw. Check the system text that was actually sent. A slider in a playground and a default in production are not the same request. The failure note puts this invention next to the other layout bugs, because it is one of them.

Top_p is another way of trimming the tail of the distribution. Using both temperature and top_p as creative knobs at once makes the route harder to reproduce and rarely fixes a prompt. Pick one place to allow variety, leave the other at the provider's default, and write the choice down. If you cannot explain why this route is not at the default, it should be at the default. Defaults are what your next incident will assume.

Penalties for repetition are a similar distraction. If the model repeats a phrase, the prompt may have asked for a long answer with little content, and the model is padding. Shorten the requested length. "Two short paragraphs" beats a repetition penalty you will forget is on. If it repeats a fact from the policy because you pasted the policy twice, delete the copy. A sampling penalty is the wrong layer. The token counter will show you the copy if you paste each half. It will not show you a temperature, because temperature is not a property of the string.

There is one legitimate reason to touch temperature during prompt work: you want to see whether the prompt is brittle. Run the same ticket five times at a moderate temperature. If you get five formats, the format line is weak, and you should strengthen the line or turn on a JSON mode, not average the formats. If you get five identical good answers, you can leave temperature low in production and trust the pilot. If you get five identical bad answers, you have a prompt bug, and lowering the setting further will not create a sixth, better answer. Brittleness tests are diagnostic. They are not a tuning hobby.

Log the setting with the prompt version. A usage review that ignores temperature will mis-blame a prompt edit that shipped in the same hour as a playground experiment someone copied into the client. The client should set the value explicitly. "Whatever the SDK defaults to this month" is not a value. SDKs change defaults. Your tests should pass the same number the production client passes. This is dull engineering and it prevents a class of ghost bugs that look like model regressions.

When a newer model arrives, rerun the twenty tickets at the same setting before you retune anything. A model change and a temperature change in one step is two changes. People do it because the new model feels different and they start sliding. Resist. First see the new model on the old prompt at the old setting. Then edit the prompt if a constraint is no longer needed or no longer enough. Then, if variety is actually part of the product, touch the sampler. The order is the discipline. The slider is the last tool, not the first, because it cannot type the sentence you never wrote.

If you want a single rule: sampling chooses among completions the prompt made plausible. Make the right completion the obvious one with a clear task, a fence around the evidence, and a constraint you can fail a test with. Then set temperature low for anything you parse, and only raise it where you asked for several different options and you have a way to pick among them. A vague prompt at a carefully chosen temperature is still a vague prompt. The bill will arrive either way.