Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IPO: Your Language Model is Secretly a Preference Classifier

Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra

TL;DR

The paper tackles the high costs of RLHF by proposing Implicit Preference Optimization (IPO), an approach that treats LLMs as implicit preference classifiers to generate training signals without external reward models. It first validates the idea by showing that LLMs can rank responses using Yes/No likelihoods on category-specific prompts and RewardBench datasets, often matching reward-model baselines. It then extends the idea into a full DSP-like pipeline through Direct Preference Optimization (DPO), constructing (Prompt, Chosen, Rejected) datasets from SFT bases and training with DPO to achieve competitive alignment results. Across multiple model families and tasks, IPO demonstrates competitive performance, with larger and instruction-tuned models benefiting more, suggesting a scalable, cost-efficient avenue for LLM alignment and self-improvement.

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. While it enables LLMs to achieve human-level alignment, it often incurs significant computational and financial costs due to its reliance on training external reward models or human-labeled preferences. In this work, we propose Implicit Preference Optimization (IPO), an alternative approach that leverages generative LLMs as preference classifiers, thereby reducing the dependence on external human feedback or reward models to obtain preferences. We conduct a comprehensive evaluation on the preference classification ability of LLMs using RewardBench, assessing models across different sizes, architectures, and training levels to validate our hypothesis. Furthermore, we investigate the self-improvement capabilities of LLMs by generating multiple responses for a given instruction and employing the model itself as a preference classifier for Direct Preference Optimization (DPO)-based training. Our findings demonstrate that models trained through IPO achieve performance comparable to those utilizing state-of-the-art reward models for obtaining preferences.

IPO: Your Language Model is Secretly a Preference Classifier

TL;DR

The paper tackles the high costs of RLHF by proposing Implicit Preference Optimization (IPO), an approach that treats LLMs as implicit preference classifiers to generate training signals without external reward models. It first validates the idea by showing that LLMs can rank responses using Yes/No likelihoods on category-specific prompts and RewardBench datasets, often matching reward-model baselines. It then extends the idea into a full DSP-like pipeline through Direct Preference Optimization (DPO), constructing (Prompt, Chosen, Rejected) datasets from SFT bases and training with DPO to achieve competitive alignment results. Across multiple model families and tasks, IPO demonstrates competitive performance, with larger and instruction-tuned models benefiting more, suggesting a scalable, cost-efficient avenue for LLM alignment and self-improvement.

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. While it enables LLMs to achieve human-level alignment, it often incurs significant computational and financial costs due to its reliance on training external reward models or human-labeled preferences. In this work, we propose Implicit Preference Optimization (IPO), an alternative approach that leverages generative LLMs as preference classifiers, thereby reducing the dependence on external human feedback or reward models to obtain preferences. We conduct a comprehensive evaluation on the preference classification ability of LLMs using RewardBench, assessing models across different sizes, architectures, and training levels to validate our hypothesis. Furthermore, we investigate the self-improvement capabilities of LLMs by generating multiple responses for a given instruction and employing the model itself as a preference classifier for Direct Preference Optimization (DPO)-based training. Our findings demonstrate that models trained through IPO achieve performance comparable to those utilizing state-of-the-art reward models for obtaining preferences.

Paper Structure

This paper contains 28 sections, 8 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Reinforcement Learning for Improving LLMs
  4. Self Improving Models
  5. Evaluation of Reward Models
  6. LLM as Preference Model
  7. Background
  8. Methodology
  9. Preference Classification
  10. Experiments
  11. Benchmarking Our Approach
  12. Comparision against Self Rewarding Approach
  13. Findings
  14. Performance of Math and Code Specific Models
  15. IPO: Implicit Preference Optimization
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Left: We evaluate preferences using (Prompt, Chosen, Rejected) triplets, scoring responses based on the probability of the token "Yes" given classification prompt. The evaluation is correct if the Chosen response scores higher than the Rejected oner. Here [PROMPT] refers to the category specific prompt. Right: Our Self-Improving DPO framework generates diverse responses, rates them, constructs a preference dataset, and trains the model via DPO.
  • Figure 2: Example outputs from Reward Bench using our approach.
  • Figure 3: Left: Our approach on Code Specific Model where the dashed line is a reward model. Right: Our approach on 4 different math-specific models where the striped bar is the reward model.