Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

TL;DR

The paper evaluates post-training pipelines for large language models under unreliable supervision, showing that SFT remains somewhat robust while RLHF via DPO struggles when demonstrations and preferences are noisy. It introduces Iterative Label Refinement (ILR), a data-centric approach that uses comparison feedback to replace poor demonstrations in the SFT dataset and retrains from scratch, enabling larger, more stable improvements than DPO under noise. Across LM-simulated and time-constrained human experiments on math, coding, and safe instruction tasks, ILR consistently outperforms SFT+DPO, with results suggesting a shift toward refining training data rather than repeatedly updating the model with noisy preferences. The work highlights practical implications for scalable AI alignment, proposing hybrid or data-first strategies to better leverage human feedback when supervision quality is imperfect.

Abstract

Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforcement learning from human feedback (RLHF). As LMs become more capable, the tasks they are given become harder to supervise. Will post-training remain effective under unreliable supervision? To test this, we simulate unreliable demonstrations and comparison feedback using small LMs and time-constrained humans. We find that in the presence of unreliable supervision, SFT still retains some effectiveness, but DPO (a common RLHF algorithm) fails to improve the model beyond SFT. To address this, we propose iterative label refinement (ILR) as an alternative to RLHF. ILR improves the SFT data by using comparison feedback to decide whether human demonstrations should be replaced by model-generated alternatives, then retrains the model via SFT on the updated data. SFT+ILR outperforms SFT+DPO on several tasks with unreliable supervision (math, coding, and safe instruction-following). Our findings suggest that as LMs are used for complex tasks where human supervision is unreliable, RLHF may no longer be the best use of human comparison feedback; instead, it is better to direct feedback towards improving the training data rather than continually training the model. Our code and data are available at https://github.com/helloelwin/iterative-label-refinement.

Figures (10)

• Figure 1: (a) In contrast to RLHF, which iteratively updates the SFT model, ILR directs comparison feedback towards improving the SFT dataset. (b) With reliable supervision, DPO effectively improves the SFT model and ILR is unnecessary. However, DPO becomes less effective with unreliable demonstrations and breaks down completely when comparison feedback is also unreliable. In contrast, ILR consistently improves on the initial SFT model with unreliable supervision, even in the most extreme case.
• Figure 2: Models finetuned on unreliable demonstrations generated by a smaller supervisor model show higher accuracy than the supervisor, demonstrating SFT's robustness to unreliable supervision. However, SFT models' full capability is only partially recovered with unreliable demonstrations. A $\to$ B denotes using demonstrations generated by the model with size A to finetune the model with size B.
• Figure 3: DPO struggles to avoid overoptimization of noisy preferences while also updating the initial suboptimal SFT model sufficiently to improve performance.
• Figure 4: ILR consistently provides more improvements for the SFT model than DPO in four settings with LM-simulated unreliable demonstrations and comparison feedback. Round 0 represents accuracy of ${\widehat{p}\,^{}_\text{SFT}}$.
• Figure 5: SFT label accuracy increases across rounds during ILR. With high-quality feedback, ILR can lead to labels that approach or even surpass the accuracy of the model trained on ground truth ($p_\text{SFT}$).