Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

Dipendra Misra, Aldo Pacchiano, Ta-Chung Chi, Ge Gao

TL;DR

The paper addresses fine-tuning LLMs with user-edit deployment data that naturally captures preferences, supervision, and costs. It proposes a principled framework (proto:learning-from-edits) with offline and online phases, deriving bounds for three learning paradigms—SFT, Direct Preference Optimization (DPO), and cost-based RL—under a balance-equation/Boltzmann-style (Bradley-Terry) model and KL-regularization via $J_\beta(\pi)$. To balance the trade-offs across feedback types, the authors introduce early ensembling and a LateEnsemble bandit strategy for online adaptation, and they demonstrate robustness to test-time user distributions. Experiments on two Gao et al. domains show the ensemble approaches outperform single-feedback methods and maintain strong transfer performance to different user models. The work provides a theoretical and practical pathway for scalable, data-efficient personalization of LLMs using naturally generated user edits.

Abstract

We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains adapted from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

TL;DR

The paper addresses fine-tuning LLMs with user-edit deployment data that naturally captures preferences, supervision, and costs. It proposes a principled framework (proto:learning-from-edits) with offline and online phases, deriving bounds for three learning paradigms—SFT, Direct Preference Optimization (DPO), and cost-based RL—under a balance-equation/Boltzmann-style (Bradley-Terry) model and KL-regularization via . To balance the trade-offs across feedback types, the authors introduce early ensembling and a LateEnsemble bandit strategy for online adaptation, and they demonstrate robustness to test-time user distributions. Experiments on two Gao et al. domains show the ensemble approaches outperform single-feedback methods and maintain strong transfer performance to different user models. The work provides a theoretical and practical pathway for scalable, data-efficient personalization of LLMs using naturally generated user edits.

Abstract

We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains adapted from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.
Paper Structure (35 sections, 24 theorems, 131 equations, 4 figures, 7 tables, 3 algorithms)

This paper contains 35 sections, 24 theorems, 131 equations, 4 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Principled Learning from User-Edits
  4. Offline Learning from User-Edits
  5. Adaptation During the Online Learning Case
  6. Theoretical Analysis of Learning from Edits
  7. Theoretical Setup and Assumptions.
  8. From Balance Equation to Bradley-Terry Distribution.
  9. Theoretical Results.
  10. Experimental Results and Discussion
  11. Conclusion and Limitations.
  12. Appendix
  13. Theoretical Analysis with Edit Distance
  14. Properties of the Balance Equation
  15. Example 1.
  16. ...and 20 more sections

Key Result

Lemma 1

For any $\pi \in \Pi$, the user distribution satisfies the following contraction property:

Figures (4)

  • Figure 1: Illustrates our learning from edits setup (proto:learning-from-edits). An interesting feature of our setup is that it contains three fundamental types of feedback that can be used to fine-tune policies. We show that using multiple feedback types leads to better policies.
  • Figure 2: Cumulative User Edits at test time corresponding to tab:main_results for the different setup. For each round, we show mean and standard deviation across 3 seeds.
  • Figure 3: Cumulative User Edits at test time corresponding to tab:main_results for the different setup. For each round, we show mean and standard deviation across 3 seeds.
  • Figure 4: Cumulative User Edits at test time corresponding to tab:transfer_results for the different setup. For each round, we show mean and standard deviation across 3 seeds.

Theorems & Definitions (26)

  • Lemma 1
  • Theorem 1: SFT Result
  • Theorem 2: DPO Result
  • Theorem 3: RL Result
  • Lemma 2: $\pi^\star$ as steady-state of user distribution
  • Lemma 2
  • Lemma 3: Optimal Policy
  • Lemma 4: TV to Unregularized Suboptimality
  • Lemma 5: TV to Regularized Suboptimality
  • Theorem 4: SFT Result Full
  • ...and 16 more