Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mechanism Design for LLM Fine-tuning with Multiple Reward Models

Haoran Sun, Yurong Chen, Siwei Wang, Xu Chu, Wei Chen, Xiaotie Deng

TL;DR

This work casts LLM fine-tuning with multiple reward models as a multi-parameter mechanism design problem, showing that SW-Max objectives alone cannot guarantee truthful preference reporting. It introduces affine maximizer payments to implement SW-Max rules in dominant strategy and proves conditions for payment equivalence and non-negativity, while also establishing approximate DSIC under input perturbations. Theoretical results are complemented by empirical studies demonstrating profitable misreporting absent payments and truthful reporting under the proposed mechanism. The proposed framework provides a principled way to price and align diverse stakeholder preferences in large-scale model fine-tuning, with practical implications for multi-reward RLHF services. The work also outlines future directions, including simpler implementability criteria beyond cycle monotonicity and extensions to broader reward-aggregation settings.

Abstract

Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply a SW-Max rule, thereby highlighting the necessity of payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for a payment rule to implement a SW-Max rule in DSIC and other principles. Third, we demonstrate that our mechanism is approximately DSIC with perturbed input, showcasing its robustness against the inevitable errors in real-world applications. Experiments on real LLM training results further confirm the practical implications of our results.

Mechanism Design for LLM Fine-tuning with Multiple Reward Models

TL;DR

This work casts LLM fine-tuning with multiple reward models as a multi-parameter mechanism design problem, showing that SW-Max objectives alone cannot guarantee truthful preference reporting. It introduces affine maximizer payments to implement SW-Max rules in dominant strategy and proves conditions for payment equivalence and non-negativity, while also establishing approximate DSIC under input perturbations. Theoretical results are complemented by empirical studies demonstrating profitable misreporting absent payments and truthful reporting under the proposed mechanism. The proposed framework provides a principled way to price and align diverse stakeholder preferences in large-scale model fine-tuning, with practical implications for multi-reward RLHF services. The work also outlines future directions, including simpler implementability criteria beyond cycle monotonicity and extensions to broader reward-aggregation settings.

Abstract

Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply a SW-Max rule, thereby highlighting the necessity of payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for a payment rule to implement a SW-Max rule in DSIC and other principles. Third, we demonstrate that our mechanism is approximately DSIC with perturbed input, showcasing its robustness against the inevitable errors in real-world applications. Experiments on real LLM training results further confirm the practical implications of our results.
Paper Structure (29 sections, 25 theorems, 124 equations, 2 figures, 6 tables)

This paper contains 29 sections, 25 theorems, 124 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Paper Organization.
  4. Preliminaries
  5. Large Language Models.
  6. Reward Modeling.
  7. Formulation of the RLHF Game
  8. Remark on the group size $\vec{w}$.
  9. Incentives in the RLHF Game
  10. Necessity of Payment Rule
  11. Affine Maximizer Payment
  12. Approximate Valuation
  13. Efficient Implementation of the Mechanism
  14. Empirical Study
  15. Models and Datasets.
  16. ...and 14 more sections

Key Result

Theorem 4.2

In the RLHF Game with mechanism $({\psi}, p)$ that ${\psi} \in \Psi^{SW}$ and $p \equiv 0$, for group $i$, define $s_i := |\{r|r = \text{rm}_i(x), x\in T^\ast\}|$ and $\underline{\text{rm}_i} := \min_{{\bm{x}} \in T^\ast} \text{rm}_i({\bm{x}})$:

Figures (2)

  • Figure 1: An illustration of the incentive issue in LLM preference aggregation. When using a basic training rule ${\psi}$ in RLHF for two groups (Alices and Bobs), fixing Bobs' report $\widetilde{\text{rm}}_2$, Alices can gain a higher utility by strategically reporting $\widetilde{\text{rm}}'_1 \neq \text{rm}_1$ than truthfully reporting $\widetilde{\text{rm}}_1 = \text{rm}_1$. On the other hand, we have $\mathrm{ASW}(\theta; \overrightarrow{\text{rm}}, \vec{w}, \theta_{\text{init}}) > \mathrm{ASW}(\theta'; \overrightarrow{\text{rm}}, \vec{w}, \theta_{\text{init}})$, which means that such strategic behavior also harms the training objective.
  • Figure 2: The simulation result for the mechanism $({\psi}, p^{AFF})$ on real LLM setup. We set the group number $n=2$, and the group size $(w_1, w_2)$ for each column is in the title. We report the valuation, the payment, and the utility for group $1$ when adopting different reporting parameters $\alpha$ and $\beta$ (defined in Section \ref{['sec:empirical']}). Truthfully reporting ($\alpha=1$ and $\beta=1$) brings the highest utility for all cases.

Theorems & Definitions (46)

  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Definition 4.1: SW-Max Training Rules
  • Theorem 4.2
  • Theorem 4.3: Simplified version of Theorem \ref{['thm:thmhigher_detailed']}
  • Proposition 4.3
  • Definition 4.4: Payment Equivalence
  • Proposition 4.4
  • Definition 4.5: Continuous Training Rule
  • ...and 36 more