Nucleus sampling (top-p)

A decoding method that picks the next token only from the smallest set of likely tokens whose probabilities add up to a chosen value p, such as 0.9.

Proposed by Holtzman and colleagues in 2019, nucleus sampling adapts the number of candidate tokens to how confident the model is, cutting off the unreliable tail of the distribution while keeping variety.

When a language model generates text it assigns a probability to every possible next token. Nucleus sampling keeps only the most probable tokens that together reach probability p, then samples among them. If the model is confident, only one or two tokens qualify; if it is uncertain, more do. See our explainer on temperature and sampling.

#Glossary #Llm #Decoding

Stories about Nucleus sampling (top-p)

Explainers

Temperature and sampling explained: why the same prompt can give different answers

Language models pick each next word from a probability distribution. Learn how temperature, top-k and top-p change it, with a worked numerical example.

thelatentdaily.com Editorial TeamOct 195 min read