Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models

Xinpeng Wang, Shitong Duan, Xiaoyuan Yi, Jing Yao, Shanlin Zhou, Zhihua Wei, Peng Zhang, Dongkuan Xu, Maosong Sun, Xing Xie

TL;DR

A detailed examination of existing alignment methods, which fall into three categories: RL-based Alignment, SFT-based Alignment, and Inference-Time Alignment, and demonstrate their intrinsic connections, strengths, and limitations are provided, to help readers better understand this research area.

Abstract

Big models have achieved revolutionary breakthroughs in the field of AI, but they might also pose potential concerns. Addressing such concerns, alignment technologies were introduced to make these models conform to human preferences and values. Despite considerable advancements in the past year, various challenges lie in establishing the optimal alignment strategy, such as data cost and scalable oversight, and how to align remains an open question. In this survey paper, we comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges. Following this foundation, we provide a detailed examination of existing alignment methods, which fall into three categories: Reinforcement Learning, Supervised Fine-Tuning, and In-context Learning, and demonstrate their intrinsic connections, strengths, and limitations, helping readers better understand this research area. In addition, two emerging topics, personal alignment, and multimodal alignment, are also discussed as novel frontiers in this field. Looking forward, we discuss potential alignment paradigms and how they could handle remaining challenges, prospecting where future alignment will go.

On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models

TL;DR

A detailed examination of existing alignment methods, which fall into three categories: RL-based Alignment, SFT-based Alignment, and Inference-Time Alignment, and demonstrate their intrinsic connections, strengths, and limitations are provided, to help readers better understand this research area.

Abstract

Big models have achieved revolutionary breakthroughs in the field of AI, but they might also pose potential concerns. Addressing such concerns, alignment technologies were introduced to make these models conform to human preferences and values. Despite considerable advancements in the past year, various challenges lie in establishing the optimal alignment strategy, such as data cost and scalable oversight, and how to align remains an open question. In this survey paper, we comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges. Following this foundation, we provide a detailed examination of existing alignment methods, which fall into three categories: Reinforcement Learning, Supervised Fine-Tuning, and In-context Learning, and demonstrate their intrinsic connections, strengths, and limitations, helping readers better understand this research area. In addition, two emerging topics, personal alignment, and multimodal alignment, are also discussed as novel frontiers in this field. Looking forward, we discuss potential alignment paradigms and how they could handle remaining challenges, prospecting where future alignment will go.
Paper Structure (24 sections, 10 equations, 7 figures)

This paper contains 24 sections, 10 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Alignment Deciphering
  3. The Trajectory of Alignment Development
  4. Alignment Formalization
  5. Value Learning
  6. Imitation Learning
  7. The Goal of Alignment
  8. Datasets and Evaluation of Alignment
  9. Datasets
  10. Evaluation
  11. The Challenges of Alignment
  12. Alignment Methods
  13. RL-based Alignment
  14. SFT-based Alignment
  15. In-Context Alignment
  16. ...and 9 more sections

Figures (7)

  • Figure 1: The genealogy of big models and their corresponding alignment methods. LMMs are marked in yellow.
  • Figure 2: The development history of value alignment.
  • Figure 3: (a) The number of papers related to alignment on ArXiv and the focused topics on each period during the past five years. (b) Visualization of paper clusters according to topics. (c) The trend of attention to each thematic category over time.
  • Figure 4: Relation between alignment method and challenges.
  • Figure 5: Illustrations of different alignment paradigms.
  • ...and 2 more figures