Prompt Layout: Order, Recency, Syntax, and the Token Budget

What this slice covers

Two prompts can contain identical words and produce different answers, because the same words arranged differently are, to a language model, a different input. This note covers the mechanics of arrangement: where instructions go, why the end of a prompt carries extra weight, how punctuation and structure make a prompt legible, when to split one request into several, and how to spend a limited context window well. The components themselves — instructions, content, examples, cues — are covered in a separate note.

Order and recency

The position of information in a prompt matters, because a model processes the input in sequence. Microsoft's research suggests putting the task first — telling the model what you want it to do before you hand over the context or examples — tends to produce better output. Their own testing adds a caveat worth knowing: with more recent chat models, some examples showed no difference at all whether the extra statement went at the beginning or at the end. So treat instructions-first as a sound default rather than a law.

The complementary effect is recency bias: material near the end of a prompt can weigh more heavily on the output than material near the start. This is why repeating the key instruction after a long block of content is worth experimenting with, and it is also why a cue works. A short phrase at the very end that primes the output — a bulleted-list opener, or the first words of the sentence you want — reliably shapes the form of what follows, because it is the last thing the model saw.

Making a prompt legible

When a prompt carries several kinds of material, mark the boundaries. Separator lines between sources or steps, section headings in uppercase, and named variables all reduce the chance that the model blurs your instruction into your data. A separator also gives you something to use as a stopping condition when you parse the output.

If you are unsure which convention to use, Markdown or XML are good bets, since models have seen enormous amounts of both. The underlying principle is that clear syntax communicates intent and makes outputs easier to parse — legibility is for the model first and for your parser second.

Breaking a task down

A long prompt that asks for one complicated thing often does worse than a sequence of prompts that each ask for one simple thing. Consider fact-checking a paragraph. Asked in one step, the model must simultaneously identify claims and invent queries. Split into two steps — first extract the factual claims, then generate a query per claim — and both halves get better, because each step has a smaller space to get wrong and you can inspect the intermediate result.

A related technique instructs the model to work through a problem step by step and show the steps, which reduces the chance of a wrong outcome and makes the answer easier to audit. Two boundaries apply. This technique is aimed at non-reasoning models; reasoning models already do this internally, and Microsoft warns that trying to extract model reasoning by means other than the supported reasoning summary is unsupported, may breach the acceptable use policy, and can result in throttling or suspension.

Parameters are part of the layout decision

Temperature and top-p both control randomness, on a scale from 0 to 2 for temperature. Lower values produce focused, concrete output — the right setting for a legal document. Higher values produce more divergent output — the right setting for fiction. The practical rule is to change one of the two at a time, not both, or you will not know which knob moved the result.

Spending the token budget

Context windows have grown, but they are still finite, and every prompt competes with itself for space. Models split text into tokens: common words are usually one token, less common words are split into pieces, and the boundaries are frequently counterintuitive — spelling a month out in full can cost fewer tokens than writing an all-numeric date.

Two habits follow. Prefer compact structures for data: a table conveys the same rows as JSON without repeating a field name on every value. And watch your whitespace — consecutive spaces are treated as separate tokens and waste budget, while a space before a word is usually absorbed into that word's token.

Mistakes people make

The first is treating layout as cosmetic and only ever editing wording. The second is stuffing every instruction, example, and document into one enormous prompt on the theory that more context is always better; past a point you are crowding out the material that actually matters. The third is tuning temperature and top-p together and then being unable to explain the result. The fourth is assuming a layout that worked for one model transfers unchanged to another — each model behaves differently, and layout is exactly where those differences show.