The LLM Interface Model
Tokens, context windows, prompts, and structured output or tool schemas can look like a grab-bag of unrelated LLM API details.
//
Search across all documentation pages
Tokens, context windows, prompts, and structured output or tool schemas can look like a grab-bag of unrelated LLM API details.
They are actually one idea: a large language model is a text-in, text-out function with a fixed-size window of attention, and every convention in this section exists to manage what goes into that window and what comes back out of it as something a program can actually use.
This page connects LLMs Basics, Tokens, Cost & Rate Limits, Prompt Engineering, and Structured Output into one model, rather than four separate API topics.
A token is the unit an LLM actually reads and writes, not quite a word and not quite a character, but a sub-word chunk produced by the model's tokenizer.
Common words are often a single token and rarer words split into several.
Every prompt you send and every response you get back is counted, priced, and limited in tokens, not in characters or words, which is why token counting is the first mental adjustment worth making.
The context window is the maximum number of tokens, prompt plus response combined in most APIs, a model can attend to in a single call, and it is a hard ceiling.
Tokens beyond it are either rejected outright or silently truncated, and nothing you do to the prompt changes that ceiling for a given model.
A useful analogy is a whiteboard with a fixed size: you can write anything you want on it, but once it's full, adding a new line means erasing an old one.
The model has no memory beyond what currently fits on that whiteboard, called from scratch on every single API call.
This leads to the central reframe of this section: the prompt is the interface.
Where a traditional API has a fixed function signature, an LLM's "signature" is whatever text you choose to place inside the context window on a given call - instructions, examples, prior conversation, retrieved documents - and the model has no access to anything outside it.
The system/user/assistant role structure is the standard shape of that interface.
The system role sets standing instructions and persona, the user role carries the actual task or question, and the assistant role holds the model's own prior responses when replaying a conversation.
Because the model is stateless between calls, a multi-turn conversation is not the model "remembering" anything.
It is your code re-sending the entire prior conversation, as tokens, inside the context window every single time, which is why conversation history has a real, growing token cost as it gets longer.
call 1: [system] [user: "hi"] -> [assistant: "hello"]
call 2: [system] [user: "hi"] [assistant: "hello"]
[user: "what's 2+2?"] -> [assistant: "4"]
Each call re-sends everything before it - the model has no state of its own between calls.
This is also why prompt design and context management are the same underlying skill: deciding what to include is deciding what fits on a whiteboard of fixed size, and every token spent on one thing is a token not available for another.
Structured output and tool/function schemas exist to solve a different problem entirely.
An LLM's native output is free-form text, but most programs need a specific, parseable shape - a JSON object with known fields, or a call to a specific function with typed arguments - to actually act on the response.
Rather than parsing free text with regexes and hoping the model's phrasing stays consistent, structured-output APIs let you supply a schema that constrains the model's response to match that shape, turning the model's words into a value your program can directly use.
Tool calling extends this same idea one step further: instead of asking the model to answer directly, you describe available functions in the prompt's schema, and the model's job becomes choosing which function to call and with what arguments.
Your code, not the model, actually executes it.
At scale, the context window's fixed size becomes a real architectural constraint rather than an inconvenience.
An application that wants a model to reason over more information than fits in one window, such as a large document, a long conversation, or a big knowledge base, can't just use a bigger prompt past that limit.
Retrieval-augmented generation exists specifically to route around this: instead of putting an entire knowledge base into every prompt, an embedding-based search retrieves only the handful of passages relevant to the current query, and only those get spent from the token budget.
Long conversations hit the same wall from a different direction, which is why real applications summarize or truncate older turns rather than replaying an ever-growing transcript verbatim into every call.
Unbounded history growth eventually exceeds the window, and even before it does, it makes every call slower and more expensive.
Structured output's guarantees also differ meaningfully by mechanism, which matters when choosing how strictly you need the shape enforced.
| Output Strategy | Strength | Weakness | Best Fit |
|---|---|---|---|
| Free-form text + manual parsing | No schema to design; maximum model expressiveness | Fragile - a phrasing change silently breaks your parser | Prototyping, human-read output, low-stakes use |
| Prompted JSON (ask nicely, parse the result) | Works with any model; no special API feature needed | The model can still emit invalid JSON or drift from the schema over long responses | Simple structured needs where occasional retries are acceptable |
| Native structured output / schema-constrained generation | Response is guaranteed or strongly constrained to match your schema | Requires provider support for the feature; constrains the model's phrasing freedom | Production systems that parse the response programmatically |
Tool schemas compound this further in agentic systems.
The model choosing the wrong tool, or supplying arguments that don't match the described types, is functionally the same failure mode as malformed JSON in plain structured output, just one layer removed - the schema is only as good as how precisely it describes what the function actually needs.
A token is a sub-word chunk produced by the model's tokenizer - common words are often one token, rarer words split into several. Models are trained and priced on tokens because that's the unit their internal vocabulary operates in, not raw characters.
The maximum number of tokens, prompt and response combined in most APIs, a model can attend to in a single call. It's a hard ceiling: content beyond it is truncated or rejected, not merely harder for the model to handle.
No, the model is stateless between calls. A multi-turn conversation works because your code resends the full relevant history as part of the prompt on every new call, which is also why long conversations cost more tokens per call as they grow.
It means the text you place in the context window - instructions, examples, history, retrieved documents - functions like a function signature. It's the entire input surface the model can act on, and nothing outside it is visible to the model at all.
They give the model a consistent way to distinguish standing instructions from the actual task at hand, and from the model's own prior turns when replaying a conversation. Without that structure, instructions and conversational content would be harder for the model to weigh correctly.
LLMs natively produce free-form text, but most programs need a specific, parseable shape to act on. Structured output lets you supply a schema that constrains the response to match that shape, instead of parsing loosely formatted text with regexes.
Structured output shapes a direct answer into a schema; tool calling has the model choose among described functions and produce arguments for one, but your own code is what actually executes that function - the model never runs anything itself.
If the document's tokens plus your instructions and expected response exceed the context window, it will be truncated or rejected outright. Even under the limit, a very large pasted document costs proportionally more per call and can dilute the model's attention on the part that actually matters.
It solves the context-window ceiling problem for large knowledge bases. Instead of trying to fit everything into every prompt, an embedding-based search retrieves only the passages relevant to the current query, so only those tokens are spent from the budget.
No. Prompted JSON can still drift from the intended schema or produce invalid JSON, especially in longer responses, while native structured-output features constrain or validate the response against your schema directly, at the cost of requiring provider support.
The entire prior conversation is resent as tokens on every call. As history grows, each subsequent call carries a larger prompt, which both costs more tokens and takes the model more time to process, independent of how simple the current question is.
Not automatically. Irrelevant, redundant, or poorly organized context can dilute the model's attention away from the part that actually matters, which is why deliberate prompt construction and retrieval tend to outperform simply including everything available.
Stack versions: This page is conceptual and describes API-shape behavior common across current LLM provider SDKs; it does not cite specific model version numbers since those change faster than this mental model does.
Reviewed by Chris St. John·Last updated Jul 19, 2026