Paper
Fine-Tuning Language Models from Human Preferences
Authors
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving
Abstract
Reward learning enables the application of reinforcement learning (RL) to
tasks where reward is defined by human judgment, building a model of reward by
asking humans questions. Most work on reward learning has used simulated
environments, but complex information about values is often expressed in
natural language, and we believe reward learning for language is a key to
making RL practical and safe for real-world tasks. In this paper, we build on
advances in generative pretraining of language models to apply reward learning
to four natural language tasks: continuing text with positive sentiment or
physically descriptive language, and summarization tasks on the TL;DR and
CNN/Daily Mail datasets. For stylistic continuation we achieve good results
with only 5,000 comparisons evaluated by humans. For summarization, models
trained with 60,000 comparisons copy whole sentences from the input but skip
irrelevant preamble; this leads to reasonable ROUGE scores and very good
performance according to our human labelers, but may be exploiting the fact
that labelers rely on simple heuristics.
Parse Issues: 4 errors