Chapter 5 — How Does AI Learn?

Separating "learning" from "using"

You may come across expressions like "the AI learned from this conversation" or "the AI is learning as it thinks right now" when talking about AI. But a model's Training and the Inference of using a trained model need to be kept separate. At the broadest level:

Training
    ↓
a trained model

a trained model
    ↓
Inference
    ↓
output such as predictions or generation

Training

Training is the process of adjusting a model's parameters or learned state based on data and a learning objective. In this process, the model is adjusted based on some learning goal. What matters here is:

Training is a separate process from giving a completed model input and obtaining output

Inference

Inference is using a trained model by giving it input to obtain predictions or output. In the scenario where we type text into a chat-style AI service and receive a reply on the spot, in terms of model use, what's happening is processing related to Inference. Inference does not mean re-learning the model's parameters in the same sense as Training.

Pretraining

For large models, the term Pretraining also comes up often. Pretraining is learning done before later adaptation to a task, domain, or instruction. Despite the name including "pre-," this does not mean Pretraining always uses the entire web, or is always unsupervised learning.

Pretraining = always uses the entire web
Pretraining = always unsupervised learning

What data is used and for what objective differs by model and method.

Fine-tuning

Fine-tuning is performing additional learning or adaptation on an already-trained model — for example, adapting a model to some task, domain, or instruction. However, Fine-tuning is not the same as Pretraining, as all of Post-training, or as necessarily Instruction Tuning.

Fine-tuning = Pretraining
Fine-tuning = all of Post-training
Fine-tuning = always Instruction Tuning

When the word Fine-tuning is used, understand it from the position of "additional learning or adaptation," and check the specific method against that material's own definition.

Post-training

Post-training is used as a collective term for additional learning or adaptation done after Pretraining. Depending on the case, it can include things such as:

• Fine-tuning
• Instruction Tuning
• preference-based adaptation
• RLHF

Which methods are included under Post-training differs by context. There's no need to think of Post-training as one fixed algorithm.

Don't equate a single conversation with Training

There's an important distinction here for using an AI service. The fact that you wrote something to the AI, the fact that that information is used in the current interaction, the fact that that information is saved, and the fact that that information causes the model's parameters to be Trained are each separate questions.

Just because the AI appeared to use information during a conversation doesn't let you conclude that the model's parameters were updated. This distinction is covered in detail in Chapter 7.

"It looks like it learned" is different from Training

Suppose you state a condition in the first half of a conversation, and the AI uses it in the second half. Seeing only that, you might be tempted to say "the AI learned in this conversation." But technically, you need to distinguish between different possibilities, such as:

• it is using the current context
• it is using saved information
• it is using retrieved information
• the model's parameters changed through Training

We don't settle on a single explanation for a mechanism that can't be confirmed from the outside.

Chapter summary

Training != Inference
Pretraining != Fine-tuning
Fine-tuning != all of Post-training
using information in a conversation != the parameters were Trained

• Training is the process of adjusting a model's parameters or learned state
• Inference is the process of using a trained model to obtain output
• Pretraining is learning done before later adaptation
• Fine-tuning is additional learning or adaptation applied to a trained model
• Post-training can include multiple adaptation methods that come after Pretraining
• you cannot conclude that Training took place just because information was used in the current conversation

The next chapter considers what kinds of work AI is prepared for, given this, and what kinds of work still require external confirmation.