How a Generative Model Actually Produces an Answer

What this slice covers

Before you can reason about which model to pick, what a deployment costs, or why one request is slower than another, you need a mental model of what happens inside a generative model between the moment your prompt arrives and the moment text comes back. This note is about that mechanism. It is deliberately not about any particular product surface — everything here is true of the models in the Foundry catalogue regardless of who built them.

The exam does not ask you to derive the mathematics. It asks you to recognise correct descriptions of the process and to spot the wrong ones, which are usually wrong in the same handful of ways.

Step one: text becomes tokens

A model does not see letters or words. Its first act is to run your text through a tokeniser, which splits it into units called tokens. A token can be a whole word, a fragment of a word, or a word plus its punctuation, depending on the tokenisation scheme the model was trained with. The full set of distinct tokens a model knows is its vocabulary, and every token in that vocabulary carries a numeric id, so a sentence becomes an ordered list of integers.

This matters more than it sounds. The tokeniser is fixed at training time. You cannot change it, and two different models will chop identical text into different numbers of pieces.

Step two: tokens become vectors

Each token id is then mapped to an embedding — a list of numbers that positions that token in a high-dimensional space. Embeddings are not arbitrary labels; they are learned during training from how tokens co-occur, so tokens used in similar contexts end up with similar vectors. This is why a model behaves as though it understands that a physician and a doctor are related, without anyone having written that rule down.

Learners often stop here and conclude that the embedding is the meaning. It is closer to say the embedding is a compressed record of statistical company a token keeps.

Step three: prediction, repeated

With the sequence represented numerically, the model weighs how much each earlier token should influence what comes next, and uses those weights together with the embeddings to predict a vector for the next position. It then picks the most probable token from its vocabulary that fits that prediction.

Then it does it again. The token it just produced is appended to the sequence, and the whole thing becomes the input for the next round. The answer is assembled one token at a time, left to right, with no plan drafted in advance and no ability to revise what has already been emitted. Microsoft's own framing compares this to autocomplete on a phone keyboard, and the comparison is exact rather than merely illustrative.

Training is the same loop run backwards. The model is shown real text, made to predict each next token while the true continuation is hidden from it, and its weights are nudged whenever the prediction misses. Nothing about that process stores facts as facts; it stores a very good approximation of which token tends to follow which.

Why identical prompts give different answers

This is the single most common source of confusion for people new to generative AI, and it follows directly from the mechanism above. Selection of the next token involves sampling, not a lookup, so a model is expected to produce fresh output rather than replay a stored response. Two consequences follow, and both appear in exam scenarios:

• Generative systems are not deterministic by default. If your requirement is reproducibility, that is something you engineer around, not something the model gives you.
• The model is not retrieving a document. When it produces a confident statement with no supporting source, that statement was generated by the same token-by-token process as everything else, which is why grounding a model in your own data is a design decision rather than a nicety.

Where the other output modalities fit

Text output is produced by the language model itself. When an application returns an image or audio instead, additional models supply the patterns for that format. Treat multimodal applications as several models cooperating, not as one model that happens to be able to draw.

Mistakes to avoid

• Describing the model as searching a database or the internet. It is not, unless the application explicitly adds retrieval on top.
• Saying the model predicts the next word. It predicts the next token, and the distinction becomes load-bearing the moment you start counting context or cost.
• Treating training and inference as the same activity. Training adjusts weights; inference only runs the prediction loop. Nothing you send at inference time changes the model.

What to carry forward

Hold three ideas: text is tokens, tokens are vectors, and generation is repeated next-token prediction. Almost every practical concern in the rest of this topic — context windows, pricing, latency, the randomness settings, why reasoning models cost more — is a direct consequence of one of those three.