Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeAL: Decoding-time Alignment for Large Language Models

James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, Dan Roth

TL;DR

DeAL introduces decoding-time alignment by treating generation as a heuristic-guided search guided by user-defined alignment objectives. It formalizes alignment as a search problem with start-state prompts and employs an A*-style search agent using start-state adaptation and top-k lookahead to enforce multiple objectives at inference. The approach yields improvements on programmatic constraints (keywords, length) and abstract aims (harmlessness, helpfulness), and can complement RLHF by enabling flexible, post-hoc alignment calibration. While promising, DeAL entails latency and access requirements, but offers a modular framework for robust, customizable alignment during decoding.

Abstract

Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the reliability of such approaches is also questionable (e.g. susceptibility to jailbreaking even after safety training). To address these issues, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints, and abstract objectives such as harmlessness and helpfulness, show that we can DeAL with fine-grained trade-offs and improve adherence to alignment objectives. Lastly, we demonstrate that DeAL is largely complementary to existing alignment strategies, and can be effectively paired with RLHF and prompting techniques to achieve better alignment.

DeAL: Decoding-time Alignment for Large Language Models

TL;DR

DeAL introduces decoding-time alignment by treating generation as a heuristic-guided search guided by user-defined alignment objectives. It formalizes alignment as a search problem with start-state prompts and employs an A*-style search agent using start-state adaptation and top-k lookahead to enforce multiple objectives at inference. The approach yields improvements on programmatic constraints (keywords, length) and abstract aims (harmlessness, helpfulness), and can complement RLHF by enabling flexible, post-hoc alignment calibration. While promising, DeAL entails latency and access requirements, but offers a modular framework for robust, customizable alignment during decoding.

Abstract

Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the reliability of such approaches is also questionable (e.g. susceptibility to jailbreaking even after safety training). To address these issues, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints, and abstract objectives such as harmlessness and helpfulness, show that we can DeAL with fine-grained trade-offs and improve adherence to alignment objectives. Lastly, we demonstrate that DeAL is largely complementary to existing alignment strategies, and can be effectively paired with RLHF and prompting techniques to achieve better alignment.
Paper Structure (31 sections, 1 equation, 1 figure, 6 tables)

This paper contains 31 sections, 1 equation, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. The Search Problem
  5. The Search Agent
  6. Start-state Adaptation
  7. Action Selection
  8. Experiments
  9. Programmatically Verifiable Objectives
  10. Keyword Constraints
  11. Length-constrained Summarization
  12. Abstract Alignment Objectives
  13. Validating Alignment Adherence
  14. Calibrating Multiple Objectives
  15. Combination with RLHF
  16. ...and 16 more sections

Figures (1)

  • Figure 1: We visualize the alignment process as a heuristic search problem during decoding based on reward functions that define various alignment objectives. Essentially, Decoding-time Alignment or DeAL lets you bring custom alignment objectives (e.g. hard/soft/logical/parametric/combination of these), and leverages them as heuristics to guide the generation path during inference.