Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans

Sky CH-Wang, Justin Svegliato, Helen Appel, Jason Eisner

TL;DR

The paper tackles the challenge of aligning LLMs with human preferences by replacing coarse A/B judgments with span-level feedback that identifies liked and disliked text portions and their rationales. It then generates incremental, left-to-right revision sequences from the base responses, turning adjacent edits into preference pairs for direct alignment. Across experiments, stepwise and cumulative rewrites yield superior model alignment and sample efficiency compared with traditional direct preference methods, despite modest annotation overhead. The approach offers a scalable, interpretable supervision framework and a dataset/tooling release to support future research in fine-grained preference optimization for retrieval-augmented generation and beyond. The work also discusses limitations, potential extensions to AI-generated feedback, and how to adapt the taxonomy to other domains and larger models, framing a path toward more efficient RLHF-style alignment.

Abstract

We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning.

Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans

TL;DR

The paper tackles the challenge of aligning LLMs with human preferences by replacing coarse A/B judgments with span-level feedback that identifies liked and disliked text portions and their rationales. It then generates incremental, left-to-right revision sequences from the base responses, turning adjacent edits into preference pairs for direct alignment. Across experiments, stepwise and cumulative rewrites yield superior model alignment and sample efficiency compared with traditional direct preference methods, despite modest annotation overhead. The approach offers a scalable, interpretable supervision framework and a dataset/tooling release to support future research in fine-grained preference optimization for retrieval-augmented generation and beyond. The work also discusses limitations, potential extensions to AI-generated feedback, and how to adapt the taxonomy to other domains and larger models, framing a path toward more efficient RLHF-style alignment.

Abstract

We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning.
Paper Structure (36 sections, 1 equation, 3 figures, 2 tables)

This paper contains 36 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Feedback Formats.
  4. Self-training.
  5. Preference Optimization.
  6. Data
  7. Response Generation.
  8. Annotation.
  9. Improvement Sequence Generation.
  10. Annotation Time.
  11. Experiments
  12. Conclusion
  13. Alignment Objective.
  14. Frontier Models.
  15. Removing the Humans.
  16. ...and 21 more sections

Figures (3)

  • Figure 1: An overview of our critique-guided improvement chain framework. A model response is annotated with span-level feedback (left) in the form of like and dislike highlights---each accompanied by labeled rationale checkboxes---and then revised step-by-step (right) into a final fully-edited response ($a^*$). Each adjacent pair ($a^{i},a^{i+1}$) in the improvement sequence is used as a preference pair for direct alignment. Blue indicates a rewrite.
  • Figure 2: Left: the original response. Right: the fully edited response $a^*$ after incorporating all user-highlighted feedback. Differences are highlighted.
  • Figure 3: Span highlighting and rationale providing annotation interface, showing the spans highlighted by a user and the provided reasons for those highlights.