Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasons to Reject? Aligning Language Models with Judgments

Weiwen Xu, Deng Cai, Zhisong Zhang, Wai Lam, Shuming Shi

TL;DR

This work investigates aligning large language models with human judgments through a novel framework called Contrastive Unlikelihood Training (CUT). CUT directly uses judgments to identify and correct inappropriate content by contrasting aligned versus misaligned contexts and applying a combination of likelihood and unlikelihood objectives. Across offline and online experiments, CUT achieves substantial gains over reward-based methods, including outperforming DaVinci003 on AlpacaEval and enabling iterative, model-specific improvement with up-to-date judgments. The results suggest that judgments can be more informative than scalar rewards for alignment, though they depend critically on judgment quality and data volume, pointing to promising directions for scalable, judgment-driven alignment. Practical impact includes improved safety and usefulness of LLMs in real-world applications, with CUT offering a scalable pathway to incorporate nuanced human judgments into model fine-tuning.

Abstract

As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval. CUT (LLaMA2-chat-13b) can also align LLMs in an iterative fashion using up-to-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval. Further analysis suggests that judgments hold greater potential than rewards in LLM alignment.

Reasons to Reject? Aligning Language Models with Judgments

TL;DR

This work investigates aligning large language models with human judgments through a novel framework called Contrastive Unlikelihood Training (CUT). CUT directly uses judgments to identify and correct inappropriate content by contrasting aligned versus misaligned contexts and applying a combination of likelihood and unlikelihood objectives. Across offline and online experiments, CUT achieves substantial gains over reward-based methods, including outperforming DaVinci003 on AlpacaEval and enabling iterative, model-specific improvement with up-to-date judgments. The results suggest that judgments can be more informative than scalar rewards for alignment, though they depend critically on judgment quality and data volume, pointing to promising directions for scalable, judgment-driven alignment. Practical impact includes improved safety and usefulness of LLMs in real-world applications, with CUT offering a scalable pathway to incorporate nuanced human judgments into model fine-tuning.

Abstract

As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval. CUT (LLaMA2-chat-13b) can also align LLMs in an iterative fashion using up-to-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval. Further analysis suggests that judgments hold greater potential than rewards in LLM alignment.
Paper Structure (24 sections, 6 equations, 9 figures, 8 tables)

This paper contains 24 sections, 6 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Setting
  5. Potential Solutions
  6. Contrastive Unlikelihood Training
  7. Incorporating Judgments for Alignment
  8. Learning from Contrasting
  9. Relation to Prior Solutions
  10. Experiments
  11. Offline Alignment
  12. Online Alignment
  13. Training A Judgment Model
  14. Judgment vs. Reward
  15. Conclusion
  16. ...and 9 more sections

Figures (9)

  • Figure 1: The illustration of three paradigms for aligning LLMs.
  • Figure 2: Generation probability of identical output text under Align-N (left) and Misalign (right) contexts.
  • Figure 3: The results of online alignment with different AI judges.
  • Figure 4: Comparison between reward-based DPO and judgment-based CUT.
  • Figure 5: The template used for aligning LLMs through CUT.
  • ...and 4 more figures