A pretrained model only continues text. Asked "Find me a charger", it may reply "and a case for my phone". Two more stages after pretraining turn it into an assistant that can call tools.

Three stages of training

StageWhat it learnsAnalogy
1 PretrainingLanguage and facts, by predicting the next wordReading the whole library
2 Supervised fine-tuningTo follow instructions and to write tool calls correctlyAn apprenticeship
3 Preference and reinforcement learningWhat people prefer, and how to succeed at tasksCoaching with feedback

Model providers run all 3 stages. Agent builders start from a model that already has them, and fine-tune only for a narrow, high-volume task.

Introduction

Supervised fine-tuning (SFT) keeps training a pretrained model on examples of (instruction, ideal response), using the same next-word loss as pretraining.

Parts of the formula:

  • is the instruction, for example "Find a 65 W charger under 50 dollars"
  • is the ideal response, for example a correct search_products tool call
  • is a penalty whenever the model finds the ideal next word unlikely

After thousands of such examples, writing a correct tool call is simply the most likely continuation.

Code

The loss is the negative log probability of the ideal tokens. A tiny example with made-up probabilities:

import numpy as np
 
# probability that the model gives to each ideal token, in order
p_ideal = np.array([0.9, 0.6, 0.8, 0.95])
 
loss = -np.sum(np.log(p_ideal))      # sum of the penalties
print(round(loss, 3))                # 0.891
 
p_ideal_after = np.array([0.99, 0.98, 0.97, 0.99])   # after more training on such examples
print(round(-np.sum(np.log(p_ideal_after)), 3))      # 0.071

Notes:

  • a probability near 1 for the ideal token gives a penalty near 0
  • in practice the library computes this as a cross-entropy loss over the tokens of the response

Copying examples teaches the format, and feedback teaches what people prefer, see Reinforcement Learning from Human Feedback (RLHF).