Chapter 4 — How Does a Model Generate Output from Input?

Shifting perspective from the AI service to the model

Chapter 3 looked at an AI service from the outside, separating what a user inputs, what output is returned, records of Search or Tools that may sometimes be observable, and internal processing that cannot be confirmed from the outside. This chapter focuses on model-based generation within that.

Here, we look at a beginner-facing conceptual flow for understanding the autoregressive language models commonly used today.

Human text does not simply flow through the model's interior as-is

We type text into an AI service. For example, the sentence "explain DNS negative caching."

But there's no need to think that the model handles this string of characters in exactly the same way a human would. The input is converted into a numerical representation the model can work with. Roughly:

That's the flow we can think of. This diagram is a conceptual diagram for understanding.

What is a Token?

A Language Model does not, as a rule, process the text a human sees as one single giant unit; instead it uses units called Tokens. A TokenA Token is the unit a model uses when handling input such as text. Depending on the tokenizer and model, a token may correspond to a whole word, part of a word, a character, or a byte — so a token and a word are not necessarily the same. → Tokenization is not necessarily identical to:

• a single character
• a single word
• a single sentence

The process of converting text into a form treated as a sequence of Tokens is called Tokenization. It's important to note here that character count and word count do not necessarily match the Token count.

character count != Token count
word count != Token count

Processing as numerical representation

A Token is bound to a numerical representation the model can compute with. What happens inside the model is not the same as a human reading text in their head — it is computation using numbers. This book does not go into the formulas or matrix operations of that computation. What matters is:

the string of characters a human handles and the representation the model handles computationally are not the same thing

Where does Transformer come in?

Transformer is a kind of neural network architecture. As we confirmed in Chapters 1 and 2:

Transformer
LLM
AI service

are not synonyms. A language model using Transformer-family architecture computes based on the input Token sequence and generates information about what Token comes next. There's no need to understand the formulas of Attention or the layer structure of Transformer here.

For Version 1, it's enough to understand:

Transformer is not "the entire AI service" — it is an architecture related to computation inside the model.

Next-token prediction

An autoregressive language model doesn't complete the entire output all at once — it generates the next Token in sequence. For example, once "DNS is" has been generated, the model computes candidates for the next Token. Once one is selected and it becomes "DNS is the name," the model uses that new state to generate the next Token.

Conceptually, this repeats:

input
  ↓
predict and select the next Token
  ↓
append the Token
  ↓
use the updated sequence
  ↓
predict and select the next Token
  ↓
...

Generation ends once a certain stopping condition is reached.

It doesn't always pick the highest-probability Token

There can be multiple candidates for the next Token. In actual generation, a Token is selected based on information related to its score or probability among the candidates. There are approaches called Sampling and Decoding for how that selection is made. So you cannot assume that "the single highest-probability Token is always chosen every time." Depending on settings and generation method, the same input can produce different outputs.

Generated text is not necessarily "a stored piece of text retrieved as-is"

Understanding this generation mechanism lets you draw one important distinction. A model generating text and simply retrieving a pre-completed, stored answer are not the same thing. Of course, an AI service as a whole may combine Search or Retrieval. But Model Generation and Search / Retrieval are kept separate here. Search and Retrieval are covered in Chapter 8.

What this explanation does not tell you

Even using this chapter's conceptual diagram, you cannot know things such as:

• a specific service's hidden prompt
• hidden reasoning
• routing
• vendor-specific backend configuration
• undisclosed model selection
• the complete internal Transformer structure

It's important not to work backward from the textbook's conceptual diagram and treat unconfirmable internal information as fact.

Chapter summary

This chapter organized model-based generation as follows:

human input
    ↓
Tokenization
    ↓
numerical representation
    ↓
computation by the model
    ↓
predicting and selecting the next Token
    ↓
feeding the generated Token back into the process
    ↓
repeat
    ↓
output

What matters is:

• human text does not simply flow through the model's interior as-is
• Tokenization converts text into a form treated as Tokens
• Transformer is a kind of model architecture
• autoregressive generation repeats the process of generating the next Token
• Token selection can involve Sampling or Decoding
• Generation and Search/Retrieval are not the same thing
• don't infer an undisclosed backend from this conceptual diagram

The next chapter looks at how such models are actually prepared in the first place, through the distinction between Training and Inference.