JJ

Generative Pre-trained Transformer (GPT)

2 min read

GPT stands for Generative Pre-trained Transformer. It is the decoder half of the Transformer, trained to predict the next token on a very large amount of plain text.

Architecture

  • GPT is a decoder only architecture
  • it removes cross-attention, because there is no encoder memory to attend to
  • what is left in each block is masked self-attention, then the feed-forward network, usually written as MLP

2 Important things to note:

  • pre-LN means layer normalisation sits before each sublayer, on the input to attention and on the input to the MLP, and the residual is added afterwards, which keeps the residual path clean and the scale stable
  • weight tying means the token embedding matrix is reused as the output classifier, so the same matrix maps tokens in and logits out

Three generations

  • GPT-1 trained on BooksCorpus, and used learned absolute position embeddings
  • GPT-2 trained on WebText, which is the primary reported data source
  • GPT-3 trained on a filtered Common Crawl mixture, and reported few-shot behaviour

The encoder side it drops is described in Transformers.

Note

What next token prediction quietly buys, with one example of each:

  • syntax, as in I put ___ fork down
  • knowledge, as in NTU is in ___
  • coreference, as in Maya called Ana because ___ was late
  • topic, as in fish, turtles, seals, and ___
  • sentiment, as in The film wasted two hours. It was ___
  • patterns, as in 1, 1, 2, 3, 5, 8, ___

Text as one interface:

  • once inputs and outputs are serialised as tokens, sentiment, translation, question answering, summarization and information extraction all become the same shape
  • the pattern is a field name, a colon, and the answer, so Review: brilliant then Label: returns positive
  • this is why one set of parameters can be pointed at many tasks with no new head

How the family moved after 2018:

  • 2018 gives GPT with pretrain then finetune, and BERT with bidirectional encoders
  • 2019 gives T5 with everything as text to text
  • 2020 gives GPT-3 with in-context task learning
  • 2022 pairs scale with alignment, so compute-optimal training and instruction following
  • 2023 gives open and visual assistants such as LLaMA, BLIP-2 and LLaVA
  • 2024 gives omni models that take text, vision and audio natively

Graph View

Backlinks