Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Human Preference Learning

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Dilek Hakkani-Tür

TL;DR

The paper introduces unsupervised human preference learning by decoupling preference modeling from the large foundation model through small, trainable preference agents that distill user preferences into natural-language rules. These rules steer a pre-trained model without modifying its weights, enabling data- and compute-efficient personalization across domains like emails, news articles, and product reviews. Empirical results from automated (GPT-4o) and human evaluations show significant improvements over baselines, with rule-based fine-tuning offering better sample efficiency than traditional I/O fine-tuning. The approach emphasizes model-specific semantic alignment, deliberative prompting, and cost considerations, while contributing new datasets and discussing licensing, release, and ethical considerations. Overall, it argues that modular preference agents enable scalable, personalized LLM applications without extensive fine-tuning of large models.

Abstract

Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.

Unsupervised Human Preference Learning

TL;DR

The paper introduces unsupervised human preference learning by decoupling preference modeling from the large foundation model through small, trainable preference agents that distill user preferences into natural-language rules. These rules steer a pre-trained model without modifying its weights, enabling data- and compute-efficient personalization across domains like emails, news articles, and product reviews. Empirical results from automated (GPT-4o) and human evaluations show significant improvements over baselines, with rule-based fine-tuning offering better sample efficiency than traditional I/O fine-tuning. The approach emphasizes model-specific semantic alignment, deliberative prompting, and cost considerations, while contributing new datasets and discussing licensing, release, and ethical considerations. Overall, it argues that modular preference agents enable scalable, personalized LLM applications without extensive fine-tuning of large models.

Abstract

Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.
Paper Structure (56 sections, 7 equations, 13 figures, 11 tables)

This paper contains 56 sections, 7 equations, 13 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Task Definition
  4. Constraints and Assumptions
  5. Model Training
  6. Model Alignment
  7. Quantifying Alignment
  8. Experimental Setup
  9. Model Choice
  10. Datasets
  11. Dataset Augmentation
  12. Synthetic Intent Generation
  13. Rule Generation
  14. Preference Agent Training
  15. Evaluation Methodology
  16. ...and 41 more sections

Figures (13)

  • Figure 1: Preference Rule Finetuning vs Naive Finetuning and Large Model Zero-Shot
  • Figure 2: On the New Yorker dataset, naive fine-tuning plateaus at a loss above 1.5, whereas fine-tuning with structured preference rules reduces the loss below 1.0 with identical hyperparameters.
  • Figure 3: Permutation of Models and Senders
  • Figure 4: Top 10 senders for the Enron-42k Dataset
  • Figure 5: Top 10 receivers for the Enron-42k Dataset
  • ...and 8 more figures