Pretraining

The first, largest stage of training, in which a model learns general patterns from huge datasets before being adapted to specific uses.

Language models are pretrained to predict the next token across trillions of tokens of text and code, absorbing grammar, facts and reasoning patterns. Pretraining is by far the most expensive stage.

Afterwards, instruction tuning and preference training turn the raw model into a helpful assistant.