Diagnosing a Refusal or a Vague Answer on an Image Prompt

The failure this slice is about

You attach a photograph of a dinner table and ask the model to plan the meal. It answers that it cannot provide that information. Nothing is broken, no error code is returned, and the model is demonstrably capable of the task. This is the most disorienting moment in early vision work, and the reflex it triggers — deploy a bigger model, or give up on the scenario — is almost always the wrong one.

Microsoft's guidance treats this as a prompting problem, not a capability verdict: when the model indicates it cannot perform a task, refine the prompt. More specific prompts guide the model toward a clearer understanding and better execution. That single reframing is the content of this slice. A refusal is diagnostic information about your instruction, and there is a repeatable procedure for reading it.

First, separate a refusal from a block

Before you rewrite anything, work out which of two very different things happened.

A refusal is the model declining in prose. The call succeeded, you were billed for it, and the assistant's message contains an apology or a statement of inability. The completion finished normally.

A content filter block is the service intervening. Content is omitted because a filter flagged it, and the response says so — the finish reason reports the filter, and Azure OpenAI's content filter results travel back in the payload.

The distinction decides your next move entirely. A refusal is answered by better prompting. A filter block is answered by examining what triggered it and, if the scenario is legitimate, by looking at filter configuration — not by rewording your way around a safety system. Teams that skip this triage waste days rewriting prompts against a block that no wording will move.

The four repair moves for a genuine refusal

Once you have established that the model itself declined, Microsoft's guidance offers four concrete adjustments, and they work best applied in order.

Ask the model what it needs. The simplest follow-up — asking what information is required — frequently unlocks the task immediately. In Microsoft's own worked example, the meal-planning refusal is followed by that question, and the model responds with a list of what it was missing: how many people are dining, dietary restrictions or preferences, the occasion, and the nutritional balance you want. Those are not obstacles; that is a specification the model has just written for you. Fold the answers back into the prompt and the task proceeds.

Put the image before the text. For single-image prompts, placing the picture ahead of the instruction is a documented improvement. The intuition is that the model establishes what it is looking at before it is told what to do about it, rather than carrying an abstract instruction into an unexamined scene.

Describe first, then act. Ask the model to describe the image in detail before completing your specific task. This splits one hard step into two easy ones, and it has a diagnostic bonus: if the description is wrong, the problem was never the task instruction. You have found a perception failure, and no amount of task rewording will fix it.

Request explanations. Asking the model to explain its generated responses converts an opaque one-line answer into something you can audit. When it explains its way to a wrong conclusion, the faulty premise is usually visible in the explanation.

Vagueness is the same disease

A bland, generic answer is a milder version of the same problem. Microsoft's contrast pairs make this vivid: "What is in this image?" against a car crash photo produces a competent but useless inventory of a silver sedan, a crumpled hood, and a utility pole. Reframed as an analysis for an auto insurance report focused on identifying and detailing damage, the same picture and the same model produce an assessment that notes the deformed bumper, the dislodged headlamp assembly, the smoke from the engine compartment, and the fact that the airbags do not appear to have deployed.

Nothing changed except the frame. The model was never withholding the detail — it had no reason to consider that detail relevant. Specificity does not squeeze more out of the model; it tells the model which of the many true things about the image you actually care about.

Common mistakes

Escalating to a larger or newer deployment as the first response to a refusal. If a one-sentence follow-up would have fixed it, the upgrade taught you nothing and costs you more forever.

Adding pressure — insisting, repeating, or asserting authority — instead of adding information. The refusals worth repairing are refusals for lack of context, and pressure supplies none.

Treating a filter block as a stubborn model. Read the finish reason before you write a single new word of prompt.

Accepting a plausible, fluent, generic answer because it did not look like a failure. A vague answer is a failed prompt that had the courtesy not to say so, and in production it is the more expensive of the two.