Tune generation behavior, such as prompt engineering and adjusting model parameters
AI-103 › Unit 2: Implement generative AI and agentic solutions › Optimize and operationalize generative AI systems › Tune generation behavior, such as prompt engineering and adjusting model parameters
Tune generation behavior, such as prompt engineering and adjusting model parameters
Tuning has two levers, and they are not interchangeable. Prompt engineering changes what the model is asked to do; parameters change how it samples and how much it produces. A surprising number of questions are decided by knowing which lever a symptom belongs to — and by two rules: adjust temperature and top_p one at a time, and do not apply non-reasoning prompt techniques to reasoning models.
Why This Matters
The documented prompt components are a checklist. Instructions, primary content, examples, cue, supporting content — a weak prompt is usually missing one, and naming which is the skill.
Recency and syntax do real work. Repeating the key instruction at the end and separating sections with clear syntax are not stylistic; they change behaviour, and the separator doubles as a stopping condition.
Two sampling parameters, one at a time. Temperature and top_p both control randomness. Changing both together makes the effect unattributable, and the documentation says to alter one at a time.
Prerequisites
- Tokens, and that both prompt and completion tokens are billed.
- That reasoning models emit hidden reasoning tokens inside the completion budget.
- Basic sampling intuition: randomness in next-token choice.
- Few-shot prompting as examples embedded in the prompt.
Learning Objectives
By the end of this lesson you will be able to:
- Compose a prompt from the five documented components.
- Apply the documented techniques and say what each fixes.
- Adjust temperature, top_p, and token limits correctly.
- Apply the reasoning model exclusions.
- Reduce token cost with the documented space efficiency guidance.
Building Blocks
The five prompt components. Instructions (what to do), Primary content (the material to act on), Examples (demonstrations of the desired behaviour), Cue (the priming start of the output), and Supporting content (extra context the model may use).
The techniques.
| Technique | Fixes |
|---|---|
| Start with clear instructions | Ambiguity about the task |
| Repeat instructions at the end | Long context burying the ask — recency bias |
| Prime the output | Wrong opening or format |
Add clear syntax (---, Markdown, XML) | Confusion between instruction and content — the separator also acts as a stopping condition |
| Break the task down | Multi-part requests done partially |
| Affordances | The model needing an external capability |
| Chain of thought | Reasoning errors — non-reasoning models only |
| Specify output structure | Unparseable responses |
| Inline citations | Unverifiable claims — better than a trailing list |
| Give the model an out | Fabrication when the answer is unavailable |
Sampling parameters. Temperature and top_p both govern randomness — and the documented guidance is to alter one at a time, not both.
Length parameters. max_tokens bounds the completion. For reasoning models the budget is max_completion_tokens / max_output_tokens, and reasoning tokens share it — reserve roughly 25,000 tokens.
Space efficiency. Tables beat JSON for the same data; consecutive whitespace becomes separate tokens; a spelled-out month is cheaper than a numeric date.
Prompt lever against parameter lever
| Attribute | ||
|---|---|---|
| Changes | What is asked | How it samples and how much |
| Fixes | Wrong content, missing constraint, bad format | Randomness, repetition, truncation |
| Signals | "It ignores a rule", "wrong format" | "Too creative", "cut off mid-sentence" |
| Rule | Not for reasoning models | One at a time |
Deep Dive
Composing from the components
A prompt that underperforms is usually missing a component rather than badly worded.
Instructions are the task. Primary content is the material — the document to summarise, the text to classify — and separating it visually from the instructions is what stops the model treating content as direction.
Examples are the most efficient way to convey format and tone. Describing a style takes a paragraph and works poorly; showing two examples takes less and works better.
Cue primes the output: ending the prompt with the beginning of the desired response — a heading, an opening bracket, a first field name — constrains the shape of what follows. It is the cheapest formatting control available.
Supporting content is context the model may draw on but is not the subject: a glossary, prior decisions, user preferences.
The diagnostic value is that a symptom maps to a missing component. Wrong tone → no examples. Wrong shape → no cue. Model treating the document as instructions → no clear syntax separating primary content.
Tuning in order
Name the symptom
Content and format problems are prompt; randomness and truncation are parameters.
The techniques that change behaviour most
Four are worth internalising because they map directly to common failures.
Repeat the key instruction at the end. Models weight recent context more heavily, so a long conversation or a large retrieved document pushes the original instruction far from the generation point. Restating it at the end is the documented remedy, and it is why "the agent followed instructions early in the conversation and drifted later" is a prompt-structure problem rather than a model one.
Add clear syntax. Separators such as ---, Markdown, or XML-style tags mark where instructions end and content begins. Two payoffs: the model is less likely to read retrieved content as direction — relevant to injection resistance — and the separator doubles as a stopping condition, giving the model a signal for where output ends.
Give the model an out. Explicit permission to say the answer is not available is the highest-leverage anti-fabrication technique. A model with no sanctioned failure response produces a confident one, because generating something is always available and admitting ignorance is not unless you allow it.
Inline citations. Documented as more reliable than a trailing reference list, because the claim and its source are generated together rather than reconstructed afterwards.
Prime the output rounds these out: ending with a cue constrains format more cheaply than describing the format at length.
Parameters, and the one-at-a-time rule
Temperature and top_p both control randomness by different mechanisms, and the documentation is explicit: alter one at a time.
The reason is attribution. Both push in the same direction, so changing both leaves you unable to say which produced the effect — and unable to reverse it precisely. Pick one as your tuning knob and leave the other at its default.
The direction is straightforward. Lower values give more deterministic, focused output: right for extraction, classification, and structured generation. Higher values give more variety: right for ideation and drafting alternatives. And a low temperature is not determinism — it reduces randomness without eliminating it, so a scenario requiring an identical output every time needs a deterministic component, not a parameter setting.
Length is separate. Truncation mid-sentence is max_tokens, not a sampling issue. And on reasoning models the trap from the deployment objective returns: reasoning tokens are billed as output and share max_completion_tokens, so a cap tuned for a non-reasoning model can be consumed entirely by thinking, producing an empty response at full cost. Reserve about 25,000 tokens.
Cost, and the space-efficiency rules
Three documented details reduce token spend without changing behaviour.
Tables beat JSON for the same data. JSON repeats every key on every record; a table names each column once. For few-shot examples and tabular context this is a large saving.
Consecutive whitespace becomes separate tokens. Indentation, blank-line padding, and aligned columns cost real tokens, so pretty-printing a large context is not free.
A spelled-out month is cheaper than a numeric date. "January 5" tokenises more efficiently than a punctuated numeric form.
Alongside these, the biggest structural saving is not sending what the model does not need — trimming retrieved context to the chunks that matter rather than passing everything retrieved. And remember reasoning tokens are billed as output, so reasoning_effort is a cost lever as much as a quality one.
Worked Examples
Example 1 — drift over a long conversation. An agent follows its formatting rule early and abandons it after many turns.
Recency bias: the instruction is far from the generation point. Repeat the key instruction at the end of the prompt, and use clear syntax so the rule is visually distinct from conversation content. Lowering temperature does not address an instruction that is effectively out of view.
Example 2 — empty responses after a model change. A summariser moved to a reasoning model returns empty messages at full cost. The team adds "think step by step".
Reasoning tokens are billed as output and share max_completion_tokens, so the existing cap was consumed by thinking. Raise it, reserving about 25,000 tokens, and tune reasoning_effort per request. Remove the chain-of-thought instruction — it is a non-reasoning technique.
Example 3 — fabrication when data is missing. An assistant invents plausible policy details when retrieval returns nothing.
Give the model an out — explicit permission and instruction to say the information is not available. Lowering temperature reduces variety, not fabrication: the model still needs a sanctioned failure response. Adding inline citations makes unsupported claims visible, and groundedness measures whether it worked.
Visual Explanations
Symptom to lever:
The reasoning-model fork:
Common Mistakes
Changing temperature and top_p together. Alter one at a time.
Treating low temperature as determinism. It reduces randomness, not variability.
Using a sampling parameter to fix a content problem.
Adding chain-of-thought to a reasoning model. A non-reasoning technique.
Reusing a non-reasoning token cap on a reasoning model.
Treating reasoning_effort as a deployment setting. It is per request.
Describing a format instead of showing examples or priming with a cue.
Omitting an out. Fabrication follows.
Pretty-printing large contexts. Consecutive whitespace costs tokens.
Practice Exercises
- Name the five prompt components and the symptom each absence produces.
- Why alter temperature and top_p one at a time, and what does a low temperature not give you?
- An instruction is followed early and abandoned later. Name the cause and the technique.
- State the reasoning-model exclusions and the parameter that replaces them.
- Give three documented ways to reduce token spend without changing behaviour.
▶Answers
- Instructions (task unclear), Primary content (model treats content as direction if not separated), Examples (wrong tone or format), Cue (wrong output shape), Supporting content (missing context it could have used).
- Because both control randomness in the same direction, so changing both makes the effect unattributable and hard to reverse — the documented guidance is to alter one at a time. A low temperature is not determinism: it reduces randomness without eliminating it, so an identical-output requirement needs a deterministic component.
- Recency bias — the instruction is far from the generation point after a long conversation or large retrieved content. Repeat the key instruction at the end, and use clear syntax so it stands apart from content.
- Prompt-engineering techniques "aren't recommended for reasoning models", and chain-of-thought prompting is explicitly non-reasoning. The lever is
reasoning_effort— a per-request parameter with valuesnone,minimal,low,medium,high,xhigh,max. - Tables instead of JSON for the same data; avoid consecutive whitespace, which becomes separate tokens; use a spelled-out month rather than a numeric date. Trimming retrieved context to what matters is the largest structural saving.
Summary & Concept Map
Tuning is two levers kept apart. The prompt lever fixes content and format, and is composed from five components — instructions, primary content, examples, cue, supporting content — with techniques that map to specific failures: repeat at the end for recency drift, clear syntax to separate instruction from content (the separator also serving as a stopping condition), prime the output for shape, inline citations over trailing lists, and give the model an out as the anti-fabrication control. The parameter lever fixes randomness and length: temperature and top_p one at a time, max_tokens for truncation, and on reasoning models a budget that reasoning tokens share. Reasoning models exclude the prompting techniques entirely and are tuned with per-request reasoning_effort. Cost falls with tables over JSON, no consecutive whitespace, and spelled-out months.
Sources and freshness
Written against current Microsoft Learn documentation for the AI-103 skills measured (16 April 2026), reviewed 2026-08-20. Microsoft Learn controls every changing product contract — availability, preview status, quotas, limits, regional support, naming, and retirement dates all move independently of this lesson. Where a scenario turns on a specific number or a GA/preview boundary, confirm it against the product's own page before relying on it.