Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora

TL;DR

The paper addresses the problem of inconsistent human annotations in preference datasets used to align language models. It introduces an automated self-curation pipeline that trains a proxy on the dataset to detect discrepancies between annotations and learned preferences, pruning inconsistent data. Across multiple learning frameworks (DPO, cDPO, rDPO) and diverse datasets, self-curation yields consistent improvements, with up to 33% gains in win scores and reduced data requirements, while remaining robust to proxy capacity. This approach offers a practical, scalable method to enhance preference learning without relying on hand-crafted heuristics, supporting more reliable alignment of language models to human preferences.

Abstract

Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33\% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ .

Figures (20)

• Figure 1: (Best viewed in color) Inconsistent annotations in preference datasets can hinder effective preference learning. To address this issue, our proposed self-curation method leverages a proxy model to identify inconsistent preferences without relying on heuristics. The method proceeds as follows: (1) Train a proxy model to score response quality using the preference dataset itself; (2) Identify discrepancies between the proxy model's predictions and the annotations in the dataset; (3) Select data that exhibit consistent preferences to enhance preference learning effectiveness.
• Figure 2: A noisy annotation from Anthropic-HH.
• Figure 3: A noisy annotation from UltraFeedback.
• Figure 4: (Best viewed in color) Preference discrepancy of UltraFeedback. We identify discrepancies between the original preference annotations made by GPT-4 and the predictions made by a proxy model trained on the given annotations. $x$-axis denotes the score difference between chosen and rejected responses, and $y$-axis indicates the number of data instances with that difference.
• Figure 5: An example from the Anthropic-HH dataset.