LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models

TL;DR

The paper addresses the challenges of standard supervised fine-tuning for large language models in reasoning tasks, especially with limited data, by introducing LaFFi, a framework where the model learns to predict the natural language feedback it would receive. LaFFi operates via a four-stage pipeline that generates predicted answers, collects AI and human feedback, trains the model to predict this feedback, and then fine-tunes with LoRA for efficiency. Experiments on SQuAD 2.0 show LaFFi improves accuracy and F1 across 3B, 7B, and 13B model scales compared to SFT and baselines, with larger models yielding larger gains. Attention analysis suggests the improvements stem from strengthened local and global token dependencies, indicating improved semantic comprehension and more human-centered responses. Overall, LaFFi demonstrates that incorporating natural language feedback into fine-tuning can reduce data requirements and enhance in-domain QA performance through a scalable, feedback-aware training paradigm.

Abstract

Fine-tuning Large Language Models (LLMs) adapts a trained model to specific downstream tasks, significantly improving task-specific performance. Supervised Fine-Tuning (SFT) is a common approach, where an LLM is trained to produce desired answers. However, LLMs trained with SFT sometimes make simple mistakes and result in hallucinations on reasoning tasks such as question-answering. Without external feedback, it is difficult for SFT to learn a good mapping between the question and the desired answer, especially with a small dataset. This paper introduces an alternative to SFT called Natural Language Feedback for Finetuning LLMs (LaFFi). LaFFi has LLMs directly predict the feedback they will receive from an annotator. We find that requiring such reflection can significantly improve the accuracy in in-domain question-answering tasks, providing a promising direction for the application of natural language feedback in the realm of SFT LLMs. Additional ablation studies show that the portion of human-annotated data in the annotated datasets affects the fine-tuning performance.

Figures (4)

• Figure 1: LaFFi is a novel Supervised Fine-Tuning approach that leverages human- and AI-generated free-form natural language feedback along with the parameter-efficient method LoRA to fine-tune LLMs, with low-data requirements.
• Figure 2: (a)&(b) Benchmarking the performance of three model scales (3B, 7B, and 13B) on three different kinds of fine-tuning modes with F1 score and accuracy. (c) X-axis: Baseline (no fine-tuning) and % of human-labeled data from {0, 20, 50, 80, 100} (with the remainder of the data coming from AI-labeled data). (d) X-axis: Baseline (no fine-tuning) and human labelled feedback (dataset sizes {200, 400, 932} rows).
• Figure 3: Heatmap visualization of the mean attention score matrix output by the last Transformer block inside LLM. The heatmap for the underlined text is displayed. Warmer colors in the heatmap correspond to higher attention scores between tokens. Regions (1) and (2) on the plots show local dependencies, while (3) shows global dependencies.
• Figure 4: An example of the training data.