Reinforcement learning from human feedback (RLHF)

A training method in which people rank model responses, a reward model learns those preferences, and the language model is optimized to score well.

RLHF made assistants far more helpful and polite; OpenAI’s InstructGPT paper (2022) described the recipe behind ChatGPT. Simpler alternatives such as Direct Preference Optimization (DPO) learn from the same comparisons without a separate reinforcement learning loop.

Read the InstructGPT paper explained and DPO explained.

#Glossary #Training #Safety

Stories about Reinforcement learning from human feedback (RLHF)

NLP & LLMsResearch

InstructGPT explained: how human feedback turned language models into assistants

The 2022 InstructGPT paper showed that reinforcement learning from human feedback makes models more helpful and less harmful. The method, the results and the trade-offs.

thelatentdaily.com Editorial TeamOct 184 min read

Reinforcement LearningResearch

PPO explained: the reinforcement learning algorithm behind RLHF

Proximal Policy Optimization made policy-gradient training simpler and more stable by limiting how far each update can move. How it works and why it is everywhere.

thelatentdaily.com Editorial TeamOct 183 min read

AI Safety & Alignment

What is AI alignment? The problem of making AI do what we intend

Why it is hard to make AI systems pursue the goals we actually want, the main techniques used today — from RLHF to constitutional AI — and the open research problems.

thelatentdaily.com Editorial TeamOct 183 min read