Examples, Decomposition, and Output Contracts in Image Prompts

From "describe this" to something a program can consume

A vision-enabled model answers in prose. That is fine for a chat window and useless for a pipeline, and the gap between the two is where most real work on this topic lives. This slice covers the three techniques that close it: showing the model the shape of the answer you want, splitting a hard request into ordered sub-goals, and stating the output format explicitly. They are independent tools, they compose, and none of them requires a different model or a single line of extra code.

Examples teach form, not facts

The most under-used technique is simply adding examples that represent the type of response you are looking for. Microsoft's worked case is deliberately mundane. Asked "what is this dog breed and what is its average weight and height?" against a photo of a border collie, the model returns a paragraph — the breed, a note about intelligence and herding, weights in pounds and kilograms, heights in inches and centimetres, and a caveat about genetics and diet. Everything in it is correct. None of it is parseable.

Now supply two prior image-and-answer pairs in the prompt, each answered in the same compact form — breed, then average weight, then average height, semicolon-separated — and then present the border collie. The model replies in exactly that form, one line, no preamble.

The crucial reading is that the examples did not teach the model anything about dogs. It already knew the answer; it produced the same facts both times. What the examples taught was the shape of a reply. This is why example selection should optimise for structural variety rather than subject-matter coverage: pick cases that pin down how you want edge cases rendered — a missing value, an ambiguous image, a multi-item answer — not cases that are merely similar to your data.

Decomposition turns one hard task into two easy ones

The second technique is to break complex requests down step by step into manageable sub-goals. Microsoft demonstrates it on a photographed recipe card. Asked what is in the image, the model transcribes the card and then hedges, because several measurements are partly obscured — an honest but unhelpful result.

The rewritten prompt names two sub-goals in sequence: first produce a list of ingredients to purchase, then produce a list of instructions to prep those ingredients. The output changes character entirely. It becomes a shopping list with practical additions the raw card never stated — which apple varieties suit baking, a reminder to check you have enough flour — followed by ordered prep steps: peel, core and slice, measure, cut the butter cold, preheat the oven.

Two things are worth noticing. First, the transcription was never the deliverable; it was an intermediate step the model had to perform anyway. Naming the real goal let it use the transcription rather than report it. Second, decomposition is also a debugging tool. When a two-stage prompt produces a wrong result, you can usually tell from the output which stage failed, and that is information a single monolithic instruction never gives you.

Declare the output format

The third technique is the most mechanical and the most neglected: clearly state the desired format for the output — markdown, JSON, HTML, and so on — and you can also specify a structure, a length, or particular attributes of the response.

Microsoft's rental-listing example combines all three constraints in one instruction: adopt the persona of an assistant that writes vacation rental listings, summarise the optimal features in three sentences, base the summary only on what is visible in the image, and close with one sentence on why the customer should book today. The result is not just better prose than "describe this image" — it is prose that fits a slot on a page. Length is bounded, the closing sentence is guaranteed, and the grounding clause suppresses invented amenities.

That grounding clause deserves separate emphasis. "Only what is visible in the image" is doing real work in a domain where models are otherwise happy to supply plausible detail. If your output feeds a listing, a report, or a record, say it.

How the three fit together

Use context and task framing to decide what the answer is about. Use decomposition to make the reasoning tractable. Use examples and a format declaration to make the result machine-readable. A mature prompt usually carries all of them: a system message that names the role and the domain, an instruction that names the sub-goals in order, one or two exemplar answers, and an explicit statement of the format and length.

Common mistakes

Choosing examples for topical similarity instead of structural coverage, so the model learns nothing about how you want unusual cases rendered.

Asking for JSON without naming the fields. "Return JSON" gets you JSON with keys of the model's own invention, which changes between calls.

Bundling five sub-goals into one sentence and then being unable to tell which one failed.

And forgetting the grounding constraint on descriptive tasks, then discovering that a generated listing confidently mentions a hot tub that does not exist.