The Image Generation Tool: Giving an Agent a Second Model

The problem the tool solves

Calling an image API directly is straightforward: your code decides an image is needed, builds a prompt, and posts it. Now imagine a conversational assistant where the user says "actually, show me what that would look like" partway through a longer exchange. Something has to notice that a picture is wanted, compose a prompt for it from the conversation so far, and weave the result back into the reply. That orchestration is exactly what the image generation tool in Foundry Agent Service provides.

The scoping guidance is admirably blunt. Use the tool when an agent needs to generate an image from a text prompt as part of a conversation or multistep workflow. Use the Azure OpenAI Image API directly for image editing, masks, or partial-image streaming. If your requirement is inpainting, the agent tool is the wrong instrument — reach for the Image API instead. The tool is currently in preview.

Two models, one project

The structural idea to internalise is that this feature needs two deployments living in the same Foundry project, and they play different roles.

The orchestrator is a chat model — a GPT-5 deployment in Microsoft's samples. It runs the conversation, decides when an image is warranted, and writes the text prompt that gets rendered. The documentation is explicit that you need a compatible orchestrator model deployed in the same Foundry project.

The image model is gpt-image-1, which the tool requires. Because it is a limited-access model, you must apply for and receive approval before you can deploy it — a lead time to plan for, not a checkbox to tick during implementation.

Setup is therefore three steps: deploy the orchestrator into your project, deploy the image model into the same project, and confirm your region and orchestrator model actually support the tool, since availability varies by both.

Attaching the tool and configuring it

You attach the tool when you define the agent version, alongside the model and the instructions. Across the SDKs the shape is consistent — an image-generation tool object carrying the image model name plus a handful of settings.

Those optional parameters are where the product decisions live. size takes a square, portrait, or landscape shape, or auto. quality takes low, medium, high, or auto. background chooses transparent, opaque, or auto. output_format selects PNG, WEBP, or JPEG, with output_compression controlling the lossy formats. And moderation sets the moderation level for the generated image, as auto or low.

One of these has an outsized effect on user experience. Generation time varies with the quality setting and prompt complexity, and Microsoft recommends low quality for time-sensitive applications. In a chat interface where a user is waiting, a fast adequate image usually beats a slow excellent one.

The routing header nobody expects

There is a mechanical detail that trips up almost every first attempt. The response request must carry a header naming your image deployment: x-ms-oai-image-generation-deployment. In the Python and TypeScript samples it is passed as an extra header; the .NET sample goes as far as implementing a custom pipeline policy purely to attach it to every request.

Without that header — or with a value that does not match the deployment name — image generation fails, and the failure is not obvious from the response. This is a fact worth memorising rather than deriving.

A related configuration hazard: the orchestrator model name in your agent definition must differ from the image generation deployment name. If they collide, the agent ends up using the wrong deployment.

Reading the result and proving it worked

The agent's Foundry model orchestrates the request and returns base64-encoded output that you save to a file. That output does not arrive as the assistant's text. It arrives as a separate output item in the response, with a type of image_generation_call and a result field carrying the base64 payload. The assistant's prose reply — something like a sentence introducing the picture — is a different item in the same output array.

So the verification procedure is precise: look for an output item whose type is image_generation_call. Alternatively, open tracing or debug for your run in the Foundry portal to confirm the tool call and inspect its inputs and outputs.

The negative case is equally diagnostic. If you see only text output and no image-generation item, the request may not have been routed to image generation at all — which sends you straight back to the header, the deployments, and the region.

Prompting and filtering

Effective prompts produce better images. Describe the subject, the visual style, and the composition you want, and use action words such as draw, create, or edit to steer the model. Content filtering can block generation when the service detects unsafe content in your prompt, so a request that produces no image is not always a configuration fault — check the filtering logs before you rewire anything.

Common mistakes

Deploying only one model and expecting the agent to render pictures by itself.

Omitting the deployment header, then debugging authentication for an hour.

Giving the orchestrator and the image deployment the same name.

Reading only the assistant's text and concluding nothing was generated.

And reaching for this tool when the real requirement is a masked edit, which belongs to the Image API.