Temperature, Top P, and the Settings That Shape a Response

What this slice covers

A deployed model is fixed — its weights, its vocabulary, its training are all settled before you ever call it. What you can still change is the request: a small set of parameters sent alongside your prompt that alter how the model turns its predictions into text. This note covers those runtime settings, what each one genuinely controls, and why the exam treats them as configuration rather than as capability.

The framing that helps most: prompts change what the model is asked; parameters change how it answers.

Temperature

Recall from the mechanics of generation that at each step the model has a distribution over possible next tokens. Temperature reshapes that distribution before a token is picked. It accepts values from 0 to 2. Raise it — around 0.7, say — and the output becomes more random and the responses more divergent. Lower it toward 0.2 and the output becomes more focused and concrete.

The documentation's own worked contrast is the one to remember: a piece of fiction is a natural fit for a higher temperature, while drafting a legal document calls for a much lower one. The generalisation is that temperature buys variety at the cost of predictability, so you spend it where variety is the product and withhold it where correctness is.

Note what temperature does not do. It does not make the model more accurate, more truthful, or more knowledgeable. A low temperature makes a wrong answer more consistently wrong, not more right.

Top P

Top probability — commonly written top_p — is the second randomness control. It also governs how varied the response is, but it reaches that result by a different route: rather than rescaling the whole distribution, it restricts selection to the most probable slice of it.

The guidance Microsoft gives here is unusually direct, and it is the single most examinable point about these two parameters: adjust one of them at a time, not both. Tuning them together means you can no longer attribute a change in behaviour to either, and the interaction is not intuitive. Pick the one you find easier to reason about, hold the other at its default, and iterate.

The parameters that sit alongside them

Two randomness dials are rarely the whole configuration. In practice a request also carries:

• A cap on generated tokens. The Chat Completions surface uses max_completion_tokens and the Responses surface uses max_output_tokens. Both bound what the model may generate, and both are your protection against a runaway answer consuming a window or a budget.
• A system or developer message. This is not a parameter in the numeric sense, but it is part of how you configure behaviour — establishing role, tone, scope, and refusal conditions before any user turn arrives.
• Streaming. Whether tokens are returned progressively or in one response changes the latency your users perceive without changing the content at all.

Which parameters a given model accepts is model-specific. Reasoning models in particular do not support the same parameter set as ordinary chat models, so code written for one family can fail against another.

How to tune without fooling yourself

Parameter tuning invites a particular kind of self-deception: you change a value, the next answer looks better, and you conclude the parameter caused it. With a sampling process, one observation proves nothing.

A defensible loop looks like this. Fix your prompt. Change exactly one parameter. Run a set of representative inputs, not a single one. Compare across the set. Only then decide. If the difference disappears when you repeat the run, you were watching randomness rather than a trend.

It is also worth checking whether the parameter is the right lever at all. Many problems that people try to solve with temperature — hallucinated details, missing structure, inconsistent formatting — are better solved by grounding the model in supplied data, specifying the output structure explicitly, or asking for citations close to the claims they support.

Mistakes to avoid

• Treating temperature 0 as a guarantee of identical output. It reduces variability substantially; it is not a determinism switch, and it does nothing about model or version changes underneath you.
• Turning both temperature and top P at once, then trying to explain the result.
• Believing lower temperature improves factual accuracy. It narrows variety, nothing more.
• Copying parameter values from a tutorial written for a different model family without checking they are supported.
• Forgetting the output cap and then being surprised by a truncated answer or an unexpected bill.

What to carry forward

Configuration lives in the request, not the model. Temperature and top P both control randomness by different means and should be tuned one at a time. Output caps, the system message, and streaming complete the picture. And when a parameter is not fixing your problem, the fault usually lies in the prompt or in the absence of grounding data, not in the dial you are turning.