How a Multimodal Model Takes an Image as Input

The shift this slice asks you to make

Most learners arrive expecting that "computer vision on Azure" means calling a purpose-built picture service: you post an image, you get back labels, boxes, or extracted text, and your code interprets a structured result. The capability Foundry asks you to understand works the other way round. There is no separate vision endpoint in this story. You call the same chat completions API you already use for text, against a model deployment that happens to accept pictures, and what comes back is prose.

Microsoft's own framing is the anchor: these are large multimodal models that analyse images and answer questions about them in natural language, combining language understanding and visual understanding inside one model. The practical consequence is liberating. Everything you know about system messages, conversation turns, token budgets, and streaming still applies. An image is not a different kind of request; it is a different kind of content inside a request you already know how to make.

Where the image actually goes in the payload

In a text-only chat call, the content of a user message is a string. In a vision call it becomes an array of content parts, and each part declares its own type. One part carries your instruction text. Another carries the image. The roles do not change — you still set a system message, you still send a user turn, and the assistant still replies with a single text answer that you read exactly as you would any other completion.

An image part can reference the picture in one of two ways, and choosing between them is the first real design decision in a vision app.

The first is a URL. The service fetches the image itself, which keeps your request body small and suits pictures that already live on the public web. The trap is that the fetch happens from Microsoft's managed infrastructure, not from inside your network. A URL your application can reach is not necessarily a URL the service can reach: private endpoints, VNet-restricted, and firewall-restricted addresses are explicitly unsupported, even when the storage account sits in the same or a peered virtual network. Teams lose hours here because the link works perfectly when pasted into a browser on a corporate machine.

The second is inline data. You base64-encode the bytes and pass them as a data URL with the MIME type in front of the payload. This is the right choice for anything private, anything generated on the fly, and anything a user has just uploaded, because nothing has to be publicly reachable. The cost is request size and an encoding step. A short helper that guesses the MIME type from the file extension and returns a data URL is the standard shape of that code.

The limits that shape what you can build

Three constraints do most of the work when you size a feature.

The format list is fixed: JPEG, PNG, GIF (first frame only), and WEBP. The GIF caveat matters more than it looks — a multi-frame animation is not a video to the model; only the first frame is seen. If your scenario is genuinely temporal, you are looking at sampling frames yourself and sending several images, not at handing over an animation.

The count limit is ten images per chat call, in the portal and through the API alike. That is a design constraint, not a quota you can raise by asking. A document with forty pages of scans cannot be a single call; it has to be batched, and you have to decide how the model carries context between batches.

The size limit is 20 MB per input image. Large phone photos routinely exceed useful sizes anyway, so most production apps downscale before sending — which is also a cost control, as the next slice on detail levels explains.

What the model returns, and what it does not

The reply is a normal chat completion. There is no bounding-box array, no confidence score, no taxonomy of tags. If you want structure, you ask for it in the prompt and parse it yourself; the model is perfectly willing to emit JSON, but it emits it as text.

Two response fields deserve attention from the first day. finish_reason tells you whether the model stopped naturally, ran out of room, or had content omitted by a filter. And the response carries content filter results specific to Azure OpenAI, so a blocked or partly blocked answer is visible in the payload rather than silently missing.

Common mistakes

The first is hunting for a dedicated image-analysis service to call. On this exam the vision path is a chat deployment, and the muscle memory of an older, task-specific image API will send you looking for endpoints that are not part of the answer.

The second is assuming a private blob URL will work because it is "in Azure". Use base64 for anything not publicly fetchable.

The third is forgetting to set a maximum token value on the completion, which quietly truncates a long visual description and makes the model look unreliable when the request was simply cut short.

The fourth is imagining that more images always means more understanding. Ten well-chosen images with a sharp instruction beat ten arbitrary ones with a vague one, every time.