Modality First: Sorting AI Workloads by What Goes In

Workload-identification questions almost never hand you a service name. They hand you a situation: a claims team drowning in supplier invoices, a broadcaster with ten thousand hours of archive footage, a support centre whose calls nobody has time to listen to. Your job is to name the kind of AI work involved. This note is about the single most reliable first move — decide what the raw input actually is before you think about anything else.

The four input modalities

Modern content processing on Azure organises itself around four kinds of input: documents, images, audio, and video. Azure Content Understanding in Foundry Tools is built to take all four and turn them into a shape you define, which is why it is the natural centre of gravity for this topic. Azure Speech in Foundry Tools is the specialist for the audio end, and the generative models in the Foundry Models catalogue sit underneath as the reasoning engine.

Each modality carries its own natural structure, and that structure is what determines which techniques even apply:

• Documents are page-structured and text-bearing. Layout matters — a number in a table cell means something different from the same number in a footer.
• Images are pixel-primary. The meaning is in what is depicted: a cracked weld, an empty shelf, a rising line on a chart.
• Audio is time-based and linguistic. Meaning arrives as a sequence, and speaker identity, language, and timing are first-class facts.
• Video is time-based and carries two channels at once, picture and sound, which is why it is normally handled as a series of segments rather than as one object.

Modality is a property of the content, not the file extension

This is where most people go wrong. A scanned PDF is a document workload even though every page is technically a photograph, because the value you want lives in the text and the layout. A phone snapshot of a receipt is likewise a document workload, not an image workload. The documentation for image analysis says this outright: image analyzers are not tuned for cases where the analysis is really about extracted text, and you should reach for a document schema instead.

Run the same test in the other direction. A frame from a security camera showing a spill on a warehouse floor is an image workload — no amount of text extraction helps, because there is no text. A thirty-second video whose visual channel is a single static album cover is, in practice, an audio workload. Ask what carries the information, not what the file is called.

Modality also fixes the limits of what is possible

Choosing the wrong modality does not just pick the wrong tool; it can promise something the technology cannot do. Video analysis samples roughly one frame per second and resizes sampled frames to a small square, so a logo that flashes for a fifth of a second, or fine print on a distant sign, may simply never be seen. Only spoken words are transcribed from a video's soundtrack — music, sound effects, and background noise are ignored. If a scenario demands frame-exact detection of a fast event, the honest answer is that a general video analyzer is the wrong instrument.

Audio has its own shape. Distinguishing who spoke which line is a real capability, as is handling a conversation that switches language partway through, and both of those are things you would never ask of a document pipeline. Documents in turn offer capabilities that make no sense elsewhere: reading checkbox states, decoding barcodes, preserving table structure across page breaks.

Scenarios that span several modalities at once

Real business processes rarely stay in one lane. An employee onboarding packet is PDF forms plus photographs of identity documents plus a recorded interview. A compliance case is email text plus a signed contract plus a call transcript. A medical claim is doctor's notes plus lab reports plus a phone consultation.

These mixed cases matter because the naive answer — pick three services and glue them together yourself — is usually the wrong one. The point of a multimodal content service is that a single analyzer can take heterogeneous inputs, reason across them, cross-reference details such as whether a name matches on every document, and emit one unified result. When a scenario stresses aggregation across formats, that is the signal.

Mistakes that cost marks

• Sorting by output instead of input. "We want a summary" tells you nothing about modality; you can summarise a contract, a call, or a video.
• Trusting the file extension. JPEG does not mean image workload, and PDF does not always mean text.
• Forgetting video is two modalities. If a question is really about what was said, the visual channel is a distraction.
• Splitting a mixed workload by hand when the scenario is explicitly about reconciling several documents against each other.

What to carry forward

Train yourself to answer three questions in order: what is the raw input, what shape must the output take, and who checks the result. Modality is the first, and it eliminates more wrong answers than any other single question you can ask.