Pretraining data is raw text collected at web scale. Curation is the cleaning step that decides what the model actually learns from.
What unlabeled means
- the text has no task labels
- it still has plenty of structure, so unlabeled does not mean unstructured
Deduplication
Removing repeated documents before training, by these named methods:
- exact hashing of normalised documents
- MinHash or locality sensitive similarity, for near duplicates
- benchmark n-gram matching, to find test items sitting in the training data
- temporal holdout by the training cutoff
Why it matters, and it is not about disk space:
- duplicates distort the training distribution, because a repeated document is effectively trained on many times
- benchmark contamination makes recall look like generalisation, so the reported score is not real
Risks from Pretraining Data
- bias
- privacy
- copyright
- freshness
- toxicity
The models trained on this data are in Generative Pre-trained Transformer (GPT).
Note
Curation sits inside a five stage pipeline, and the order matters.
- collect web pages, books, code and papers
- clean by decoding, stripping boilerplate and segmenting
- filter on language, quality, safety and policy
- dedup to remove exact and near duplicate copies
- prepare by tokenizing and packing into windows
Three things every corpus has to document:
- coverage, so which languages, domains, genres and time range are present
- provenance, so the source, the licence and the timestamps
- holdouts, so the validation and test material and the benchmark decontamination
A corpus is not fed in raw proportions. It is sampled from a mixture:
- each source carries a weight, and the weights are a probability distribution over corpora
- a document is drawn by first choosing a corpus and then choosing a window inside it
- the weights decide how many gradient updates each domain receives, so a small clean source can be lifted and a huge noisy source pushed down
Three splits, three jobs:
- train fits the parameters
- validation chooses settings such as model size, learning rate and stopping
- test is reported once, on unseen samples