Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D. Yao, Shi-Xiong Zhang, Sambit Sahu

TL;DR

The objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners and to encourage further engagement and innovation in this area.

Abstract

Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

TL;DR

The objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners and to encourage further engagement and innovation in this area.

Abstract

Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.
Paper Structure (113 sections, 74 equations, 4 figures, 12 tables, 3 algorithms)

This paper contains 113 sections, 74 equations, 4 figures, 12 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Tasks and Definition
  4. RL Framework Concepts
  5. Policy Model
  6. Reward Model
  7. Action Space
  8. Environment
  9. Preference Data
  10. Terminology and Notation
  11. Taxonomy
  12. Sampling
  13. Modality
  14. Language
  15. Reward Granularity
  16. ...and 98 more sections

Figures (4)

  • Figure 1: Taxonomy of the Preference Tuning methods.
  • Figure 2: Preference Tuning methods. The circles with shaded areas represent off-policy methods, while the unshaded circles denote on-policy methods. The overlapping area signifies methods that incorporate both on-policy and off-policy approaches. The policy-agnostic circle indicates methods that are applicable to either on-policy or off-policy scenarios. The combination circle represents methods that integrate both online and off-policy strategies.
  • Figure 3: Training stages.
  • Figure 4: Preference Tuning methods for online algorithms, such as RLHF, Online DPO, and SFT-like, and offline methods, such as DPO.