Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Multi-Turn Preference Optimization for Language Agents

Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, Fuli Feng

TL;DR

The paper addresses the difficulty of adapting LLMs to multi turn language agent tasks by extending direct preference optimization through a state action occupancy based constraint and length normalization. It introduces the Direct Multi-Turn Preference Optimization DMPO loss which eliminates the partition function in the Bradley-Terry model and reweights trajectory contributions to emphasize early, high quality actions. Theoretical analysis clarifies why length normalization cancels the partition function and explains the gradient dynamics, while extensive experiments on WebShop, ScienceWorld, and ALFWorld show that DMPO outperforms single turn DPO and other baselines, including in noisy settings. The work provides a practical, robust objective for agent fine tuning and offers insights into the role of trajectory length and early actions in preference based RL for language agents.

Abstract

Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss. The code is available at https://github.com/swt-user/DMPO.

Direct Multi-Turn Preference Optimization for Language Agents

TL;DR

The paper addresses the difficulty of adapting LLMs to multi turn language agent tasks by extending direct preference optimization through a state action occupancy based constraint and length normalization. It introduces the Direct Multi-Turn Preference Optimization DMPO loss which eliminates the partition function in the Bradley-Terry model and reweights trajectory contributions to emphasize early, high quality actions. Theoretical analysis clarifies why length normalization cancels the partition function and explains the gradient dynamics, while extensive experiments on WebShop, ScienceWorld, and ALFWorld show that DMPO outperforms single turn DPO and other baselines, including in noisy settings. The work provides a practical, robust objective for agent fine tuning and offers insights into the role of trajectory length and early actions in preference based RL for language agents.

Abstract

Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss. The code is available at https://github.com/swt-user/DMPO.
Paper Structure (33 sections, 2 theorems, 17 equations, 5 figures, 4 tables)

This paper contains 33 sections, 2 theorems, 17 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-Context Learning
  4. Agent Tuning
  5. Preference-Based RL
  6. Preliminary
  7. Task Description
  8. Direct Preference Optimization
  9. Method
  10. State-Action Occupancy Measure
  11. DMPO
  12. In-Depth Analysis
  13. Length Normalization Explanation
  14. Further Discussion
  15. Experiments
  16. ...and 18 more sections

Key Result

Corollary 4.0.1

The DMPO loss assigns higher weights to state-action pairs at early steps, where the weight is related to discount factor $\gamma$.

Figures (5)

  • Figure 1: Illustration of DMPO loss, which directly optimizes the RL objective by maximizing the likelihood of the preferred trajectory over the dispreferred trajectory.
  • Figure 2: Illustration of expert trajectories and trajectories learned under the constraints of policy and state-action occupancy measure.
  • Figure 3: The effect of hyperparameter $\gamma$ on the relative performance of the model trained with DMPO loss on the WebShop dataset in both noisy and clean settings.
  • Figure 4: The effect of "loss" trajectories length on the performance of the model trained with DPO and DMPO loss in the noisy setting on ScienceWorld. The base model is Mistral-7B-Instruct-v0.2.
  • Figure 5: Case study of WebShop.

Theorems & Definitions (4)

  • Corollary 4.0.1
  • proof
  • Corollary 4.0.2
  • proof