A Survey on Human Preference Learning for Large Language Models

TL;DR

<p>This survey anchors human preference learning as a central approach to aligning large language models with human intentions, detailing how feedback sources, formats, and modeling choices shape preference signals and their downstream use. It synthesizes three main threads—numerical and linguistic preference modeling, and implicit signaling—across RLHF, supervised fine-tuning, and contrastive methods, while evaluating alignment via open-form, automatic, and qualitative benchmarks. The work further identifies practical challenges and promising directions, such as scalable oversight, language-agnostic and multi-modal alignment, and robust evaluation, to guide future research in aligning LLMs with diverse human values. The synthesis emphasizes the necessity of combining high-quality feedback with scalable methods and careful evaluation to ensure reliable, safe, and beneficial LLM behavior across contexts.

Abstract

The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning, enhancing LLMs with excellent applicability and effectiveness in a wide range of contexts. Despite the numerous related studies conducted, a perspective on how human preferences are introduced into LLMs remains limited, which may prevent a deeper comprehension of the relationships between human preferences and LLMs as well as the realization of their limitations. In this survey, we review the progress in exploring human preference learning for LLMs from a preference-centered perspective, covering the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs. We first categorize the human feedback according to data sources and formats. We then summarize techniques for human preferences modeling and compare the advantages and disadvantages of different schools of models. Moreover, we present various preference usage methods sorted by the objectives to utilize human preference signals. Finally, we summarize some prevailing approaches to evaluate LLMs in terms of alignment with human intentions and discuss our outlooks on the human intention alignment for LLMs.

Figures (3)

• Figure 1: Overview of the aspects of human preference learning included in this survey.
• Figure 2: A timeline of the milestones in the development of human preference learning for LLMs.
• Figure 3: Illustrations of different preference usage paradigms, exemplified by LLM alignment.