Prompting Vision-Enabled Chat Models on Images

The other half of the vision area is the reverse of image generation: a model that reads a picture. A vision-enabled chat model takes an image together with text and answers in language, which means the skill you need is not configuring a vision API but writing a prompt. The same photograph can yield a flat inventory of objects or a usable insurance assessment depending entirely on what you asked for. These techniques apply to vision-enabled chat models such as GPT-4 Turbo with Vision, GPT-4o and GPT-4o-mini, and you can rehearse all of them in the Foundry playground by editing the system prompt and resending the same image.

The default answer is the least useful answer

Ask a vision model "what is in this image?" and you get a competent, neutral description. Shown a campsite it will report a yellow tent, two folding chairs in red and blue, a portable table and surrounding trees. Everything is correct. Nothing is useful, because the model had no idea what the answer was for.

That gap between accurate and useful is the entire subject of this note. Every technique below is a way of telling the model what job the description is doing, so it can choose which details matter and which to drop.

Contextual specificity

Adding context to the scenario helps the model judge what an appropriate output looks like, and it also tells the model what to leave out. Asked to describe the same campsite for an outdoor hiking product catalogue, with enthusiasm and professionalism, the model stops listing objects and starts writing marketing copy: the tent becomes spacious and durable, the chairs become lightweight and eye-catching, and the whole thing reads as a product blurb.

Same image, same model, different world. The context did not add information — it selected which information mattered.

Task-oriented prompts

Closely related, but sharper: give the model a job rather than a setting. A photograph of a car crashed into a utility pole produces, by default, a fair description of a damaged silver sedan. Reframe the request as an assessment for an auto insurance report focused on identifying and detailing damage, and the response reorganises itself around what an assessor needs — the crumpled hood, the deformed and partly detached bumper, the dislodged headlamp assembly, smoke from the engine compartment implying possible mechanical or electrical damage, and the observation that the airbags do not appear to have deployed.

Notice what happened to the background. Houses and hedges were prominent in the neutral description and vanish from the insurance one. Focus is subtraction as much as addition.

Handling refusals as a prompt problem, not a wall

Sometimes the model declines. Shown a table of food and told to "plan this meal", it may simply say it cannot provide that information. The instinct is to conclude the model cannot do the task. Usually the prompt was underdetermined rather than the task impossible.

The productive move is to refine and continue. Asking the model what information it needs turns a refusal into a specification: how many diners, what dietary restrictions apply, what kind of occasion, what nutritional balance is wanted. A few other tactics help for the same reason — ask the model to explain its responses, place the image before the text when there is a single image, and have it narrate the picture thoroughly as a first step, before it attempts your actual task. Each gives the model a firmer footing before the hard part.

Examples, decomposition and output format

Three techniques remain, and they are the ones that make output predictable enough to put in a pipeline.

Adding examples shows the model the shape of answer you want. Asked about a dog's breed and typical size, an unguided model returns a paragraph of prose. Precede the question with two image-and-answer pairs formatted tersely — breed, average weight, average height — and the model answers the new image in exactly that form. You never described the format; you demonstrated it.

Breaking down requests turns one complex instruction into manageable sub-goals. A photographed recipe card, read plainly, yields a transcription with gaps where measurements were unclear. Recast as two ordered tasks — produce a shopping list, then produce preparation instructions — and the same image yields something a person could act on.

Defining the output format is the simplest lever and the most neglected. State the format you want, such as markdown, JSON or HTML, and you may also specify structure, length or particular attributes of the response. Asking for a rental listing summarised in three sentences, based only on what is visible in the image, and ending with one sentence on why to book today, produces exactly that — a constraint on length, a constraint on grounding, and a required closing move, all honoured.

What to carry into practice

Three ideas are worth holding. First, prompt quality, not model choice, explains most of the variation you will see between a mediocre and an excellent result. Second, grounding instructions matter with images: telling the model to rely only on what is visible is how you discourage plausible invention. Third, treat refusals and vague answers as feedback on your prompt, and iterate — that iterative loop is the actual skill being tested.