A pretrained model only continues text. Asked "Find me a charger", it may reply "and a case for my phone". Two more stages after pretraining turn it into an assistant that can call tools.
Three stages of training
| Stage | What it learns | Analogy |
|---|---|---|
| 1 Pretraining | Language and facts, by predicting the next word | Reading the whole library |
| 2 Supervised fine-tuning | To follow instructions and to write tool calls correctly | An apprenticeship |
| 3 Preference and reinforcement learning | What people prefer, and how to succeed at tasks | Coaching with feedback |
Model providers run all 3 stages. Agent builders start from a model that already has them, and fine-tune only for a narrow, high-volume task.
Introduction
Supervised fine-tuning (SFT) keeps training a pretrained model on examples of (instruction, ideal response), using the same next-word loss as pretraining.
Parts of the formula:
- is the instruction, for example "Find a 65 W charger under 50 dollars"
- is the ideal response, for example a correct
search_productstool call - is a penalty whenever the model finds the ideal next word unlikely
After thousands of such examples, writing a correct tool call is simply the most likely continuation.
Code
The loss is the negative log probability of the ideal tokens. A tiny example with made-up probabilities:
import numpy as np
# probability that the model gives to each ideal token, in order
p_ideal = np.array([0.9, 0.6, 0.8, 0.95])
loss = -np.sum(np.log(p_ideal)) # sum of the penalties
print(round(loss, 3)) # 0.891
p_ideal_after = np.array([0.99, 0.98, 0.97, 0.99]) # after more training on such examples
print(round(-np.sum(np.log(p_ideal_after)), 3)) # 0.071Notes:
- a probability near 1 for the ideal token gives a penalty near 0
- in practice the library computes this as a cross-entropy loss over the tokens of the response
Copying examples teaches the format, and feedback teaches what people prefer, see Reinforcement Learning from Human Feedback (RLHF).