Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

Saaduddin Mahmud, Mason Nakamura, Shlomo Zilberstein

TL;DR

MAPLE addresses the challenge of learning human preferences for sequential decision making by combining natural language feedback with conventional trajectory rankings in a Bayesian framework guided by large language models. It represents the preference function $\omega(\tau)$ as a linear combination of language based concepts and uses a language conditioned active query strategy to select informative and easy queries, reducing human burden. The method updates the posterior over $\omega$ via MCMC using both a Bradley Terry likelihood from trajectory rankings and LLM generated priors from linguistic feedback, enabling interpretable auditing of the learning process. Experiments on an OpenStreetMap based routing benchmark and a Minigrid style HomeGrid environment show improved sample efficiency and alignment with human preferences, highlighting MAPLE's potential for real world route planning and household robot tasks.

Abstract

The advent of large language models (LLMs) has sparked significant interest in using natural language for preference learning. However, existing methods often suffer from high computational burdens, taxing human supervision, and lack of interpretability. To address these issues, we introduce MAPLE, a framework for large language model-guided Bayesian active preference learning. MAPLE leverages LLMs to model the distribution over preference functions, conditioning it on both natural language feedback and conventional preference learning feedback, such as pairwise trajectory rankings. MAPLE also employs active learning to systematically reduce uncertainty in this distribution and incorporates a language-conditioned active query selection mechanism to identify informative and easy-to-answer queries, thus reducing human burden. We evaluate MAPLE's sample efficiency and preference inference quality across two benchmarks, including a real-world vehicle route planning benchmark using OpenStreetMap data. Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.

MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

TL;DR

MAPLE addresses the challenge of learning human preferences for sequential decision making by combining natural language feedback with conventional trajectory rankings in a Bayesian framework guided by large language models. It represents the preference function as a linear combination of language based concepts and uses a language conditioned active query strategy to select informative and easy queries, reducing human burden. The method updates the posterior over via MCMC using both a Bradley Terry likelihood from trajectory rankings and LLM generated priors from linguistic feedback, enabling interpretable auditing of the learning process. Experiments on an OpenStreetMap based routing benchmark and a Minigrid style HomeGrid environment show improved sample efficiency and alignment with human preferences, highlighting MAPLE's potential for real world route planning and household robot tasks.

Abstract

The advent of large language models (LLMs) has sparked significant interest in using natural language for preference learning. However, existing methods often suffer from high computational burdens, taxing human supervision, and lack of interpretability. To address these issues, we introduce MAPLE, a framework for large language model-guided Bayesian active preference learning. MAPLE leverages LLMs to model the distribution over preference functions, conditioning it on both natural language feedback and conventional preference learning feedback, such as pairwise trajectory rankings. MAPLE also employs active learning to systematically reduce uncertainty in this distribution and incorporates a language-conditioned active query selection mechanism to identify informative and easy-to-answer queries, thus reducing human burden. We evaluate MAPLE's sample efficiency and preference inference quality across two benchmarks, including a real-world vehicle route planning benchmark using OpenStreetMap data. Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.

Paper Structure

This paper contains 33 sections, 8 theorems, 10 equations, 14 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning from demonstration
  4. Natural language in intention communication
  5. Abstraction in reward learning
  6. Alignment auditing
  7. Active learning
  8. Background
  9. Markov decision process (MDP)
  10. Bayesian preference learning
  11. Variance ratio
  12. Problem Formulation
  13. MAPLE
  14. Motivating example
  15. Detailed Description of the Proposed Method
  16. ...and 18 more sections

Key Result

Proposition 1

Assuming the independence of AQSR from acquisition function ranking, the QSR of a random query selection strategy: $P(q \in Q_{\mathcal{A}} \mid \text{random}) = AQSR$

Figures (14)

  • Figure 1: Application of MAPLE to the Natural Language Vehicle Routing Task.
  • Figure 2: OpenStreetMap Routing
  • Figure 3: HomeGrid
  • Figure : (a) Test accuracy (OSM Routing)
  • Figure : (a) Test accuracy (OSM Routing)
  • ...and 9 more figures

Theorems & Definitions (11)

  • Definition 1
  • Definition 2
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Corollary 1
  • Definition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • ...and 1 more