Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Suppressing Pink Elephants with Direct Principle Feedback

Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, Stella Biderman

TL;DR

The paper tackles inference-time controllability for LLMs by formalizing the Pink Elephant Problem, where models must refrain from discussing a specified entity. It introduces Direct Principle Feedback (DPF), a streamlined RLAIF approach that uses response–revision pairs and Direct Preference Optimization to imbue models with novel behavioral constraints without hard-coding. Through a 162K-dialogue synthetic dataset spanning 29 domains, the authors demonstrate that DPF-trained models achieve GPT-4 level performance on pink-elephant avoidance while preserving broader capabilities. They provide extensive qualitative and quantitative analyses, including human/GPT-4 annotations and MT-Bench scores, and discuss ethical considerations and future directions toward flexible safety training and generalization to other failure modes.

Abstract

Existing methods for controlling language models, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a language model. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a ``Pink Elephant''), and instead discuss a preferred entity (``Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.

Suppressing Pink Elephants with Direct Principle Feedback

TL;DR

The paper tackles inference-time controllability for LLMs by formalizing the Pink Elephant Problem, where models must refrain from discussing a specified entity. It introduces Direct Principle Feedback (DPF), a streamlined RLAIF approach that uses response–revision pairs and Direct Preference Optimization to imbue models with novel behavioral constraints without hard-coding. Through a 162K-dialogue synthetic dataset spanning 29 domains, the authors demonstrate that DPF-trained models achieve GPT-4 level performance on pink-elephant avoidance while preserving broader capabilities. They provide extensive qualitative and quantitative analyses, including human/GPT-4 annotations and MT-Bench scores, and discuss ethical considerations and future directions toward flexible safety training and generalization to other failure modes.

Abstract

Existing methods for controlling language models, such as RLHF and Constitutional AI, involve determining which LLM behaviors are desirable and training them into a language model. However, in many cases, it is desirable for LLMs to be controllable at inference time, so that they can be used in multiple contexts with diverse needs. We illustrate this with the Pink Elephant Problem: instructing an LLM to avoid discussing a certain entity (a ``Pink Elephant''), and instead discuss a preferred entity (``Grey Elephant''). We apply a novel simplification of Constitutional AI, Direct Principle Feedback, which skips the ranking of responses and uses DPO directly on critiques and revisions. Our results show that after DPF fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on our curated test set assessing the Pink Elephant Problem.
Paper Structure (40 sections, 6 figures, 2 tables)

This paper contains 40 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Inference-Time Controllability via Direct Principle Feedback
  4. Simplifying RLAIF with Direct Principle Feedback
  5. Applying DPF to the Pink Elephant Problem
  6. Dataset Generation
  7. Topics Generation
  8. Pink Elephant Pairs
  9. Unwanted Behavior Generation
  10. Revised Behavior Generation
  11. Data Cleaning
  12. Experiments
  13. Training
  14. Baselines
  15. Quantitative Evaluation
  16. ...and 25 more sections

Figures (6)

  • Figure 1: This chatbot is designed to help British students apply to British universities, but doesn't have up-to-date information about other universities. "American Universities" is the Pink Elephant while "British Universities" is the Grey Elephant.
  • Figure 2: A high-level illustration of Constitutional AI bai2022constitutional, RLAIF without critiques, and DPF (ours). DPF streamlines the process to use only a single step of AI feedback, and uses revisions according to a principle as feedback. Note that the choice of objective such as DPO is arbitrary and may be substituted.
  • Figure 3: Initial dataset generation step, as described in \ref{['sec:topic-gen']} and \ref{['sec:pep-pairs']}. A short list of topic areas is handwritten, GPT-4 is prompted to generate many contrasting Pink Elephant entity pairs, and StableBeluga2 subsequently prompted to create a number of plausible subtopics for each entity pair. Duplicates are removed at each step.
  • Figure 4: Generation of \ref{['sec:undesired-datagen']} dialogues which exhibit undesired behavior (mentioning the Pink Elephant in the final turn). We perform an intermediate planning step.
  • Figure 5: Critique and Revision step, as described in \ref{['sec:revision-datagen']}. We ask our model to rewrite the final dialogue turn to remove mentions of the Pink Elephant.
  • ...and 1 more figures