Tokens and the Context Window: The Budget Every Request Spends

What this slice covers

Tokens are the unit of account for everything a generative model does. They determine how much you can send, how much comes back, what you pay, and how fast you are allowed to go. This note treats the token and the context window as one connected idea, because almost every practical limit you meet in Microsoft Foundry is expressed in them.

If you only remember one thing: a request is not measured in characters, sentences, or pages. It is measured in tokens, and the number is not one you can guess reliably.

Vocabulary and tokenisation

A tokeniser converts text into the discrete units a model was trained on, and the complete set of units it knows is called its vocabulary. The splitting method varies by model. Word tokenisation cuts on delimiters; character tokenisation produces one token per character; subword tokenisation cuts words into fragments. Subword schemes dominate modern language models, and byte-pair encoding is the specific variety the GPT family uses.

The trade-off is worth internalising because it explains a lot of otherwise puzzling behaviour. Smaller tokens let a model cope with typos, rare words, and unusual syntax, and keep the vocabulary compact — but the same passage becomes more tokens, so it costs more compute and eats more of your budget. Larger tokens are cheaper per passage but force a bigger vocabulary and handle unknown words badly.

Two practical consequences: token counts are counterintuitive (a fully numeric date can consume more tokens than spelling the month out), and consecutive spaces become separate tokens, so sloppy whitespace quietly costs money.

The context window

Every model has a ceiling on how many tokens it can work with. In current documentation this ceiling is normally stated as a single combined figure spanning both what you send and what the model generates.

That word combined is where learners trip. The window is not an input allowance with a separate output allowance beside it. Input and output share it. If you fill 95 percent of a window with a long document, the model has almost nothing left to answer with, and you will get a truncated response rather than an error telling you the prompt was too big. Planning a request therefore means budgeting: how much context do I need to supply, and how much room must I reserve for the answer?

Window sizes differ from model to model, and a newer version of the same model family may change the figure. Never hard-code an assumption about capacity that you have not checked against the specific model you deployed.

Tokens as currency and as throughput

The token does double duty beyond the window.

First, it is the billing unit for pay-per-token deployments. You are charged on input and output tokens, and the two are usually priced differently — output tends to be the more expensive side, which is why asking for terse answers is a genuine cost lever rather than a style preference.

Second, it is the throttling unit. Rate limits in Foundry are expressed largely in tokens per minute alongside requests per minute. So the same design choice — stuffing a large document into every prompt — simultaneously shrinks your room to answer, raises your bill, and brings you closer to being rate-limited. Three problems, one cause.

How this shapes design

Once you see tokens as a shared budget, several standard practices stop looking like folklore:

• Summarise or chunk long source material rather than pasting it whole, because the window is finite and the reserved output space is not optional.
• Prefer compact data formats in prompts. Tabular layouts carry the same information as verbose key-per-field structures for fewer tokens.
• Trim conversation history deliberately in multi-turn applications. Each turn resends what came before, so an unmanaged chat grows its own cost curve.
• Measure rather than estimate. Counting tokens with a tokeniser for your actual model is the only reliable method.

Mistakes to avoid

• Equating a token with a word. Common words often are single tokens, but rarer words split into several, and that ratio varies by language.
• Believing the input limit and output limit are independent. When a combined window is quoted, they compete for the same space.
• Assuming a bigger context window removes the need to be selective. A larger budget is still a budget, and every token you add is billed and counted against your rate limit.
• Reasoning about a model's capacity from another model's numbers. Both the window size and the tokenisation method are model-specific.

What to carry forward

Tokens are how a model reads, how Foundry meters you, and how it throttles you. The context window is a shared ceiling over input and output together. Hold those two facts and the pricing, quota, and deployment material later in this topic reads as one consistent system rather than a set of unrelated rules.