Anyprefer: An Agentic Framework for Preference Data Synthesis

TL;DR

The paper addresses the challenge of producing high-quality preference data for aligning foundation models with human values, noting biases and scalability issues in self-rewarding approaches. It introduces Anyprefer, a framework that casts preference data synthesis as a cooperative two-player Markov game between a Target Model and a Judge Model, augmented with external tools and a reward model, plus a feedback loop and iterative Direct Preference Optimization. The key contributions are the tool-augmented judge mechanism, the reward-guided data quality evaluation, the feedback-driven prompt optimization, and the Anyprefer-V1 dataset containing 58K high-quality preference pairs across four domains with strong empirical gains on 21 datasets. This work demonstrates scalable, cross-domain preference data synthesis that improves model alignment in natural language generation, vision-language understanding, medical image analysis, and visuo-motor control, offering practical benefits for alignment research and open-source data resources.

Abstract

High-quality preference data is essential for aligning foundation models with human values through preference learning. However, manual annotation of such data is often time-consuming and costly. Recent methods often adopt a self-rewarding approach, where the target model generates and annotates its own preference data, but this can lead to inaccuracies since the reward model shares weights with the target model, thereby amplifying inherent biases. To address these issues, we propose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model. Anyprefer frames the data synthesis process as a cooperative two-player Markov Game, where the target model and the judge model collaborate together. Here, a series of external tools are introduced to assist the judge model in accurately rewarding the target model's responses, mitigating biases in the rewarding process. In addition, a feedback mechanism is introduced to optimize prompts for both models, enhancing collaboration and improving data quality. The synthesized data is compiled into a new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs. Extensive experiments show that Anyprefer significantly improves model alignment performance across four main applications, covering 21 datasets, achieving average improvements of 18.55% in five natural language generation datasets, 3.66% in nine vision-language understanding datasets, 30.05% in three medical image analysis datasets, and 16.00% in four visuo-motor control tasks.

Figures (8)

• Figure 1: The figure illustrates the Anyprefer framework. First, Anyprefer selects the necessary tools based on the input prompt to obtain supplementary information, which is then integrated into a knowledge base. Next, the target model generates several responses for the input data. The judge model then ranks these responses using the constructed knowledge base. Subsequently, Anyprefer combines the best and worst-ranked responses into a preference pair. The reward model will then evaluate the quality of this preference pair, and all unqualified pairs will go through the optimization stage to refine its quality by using the proposed feedback mechanism.
• Figure 2: We evaluated Anyprefer using benchmarks from four applications. The target model represents the original model before preference fine-tuning. For medical image analysis, "B" for BLEU, "R" for ROUGE-L, "M" for METEOR, "C" for closed, and "O" for open tasks. In medical iamge analysis, "RAD": VQA-RAD, "IU": IU-Xray.
• Figure 3: The performance of Anyprefer at different iterations over all applications.
• Figure 4: Impact of tools (T) and feedback (F) on judge model.
• Figure 5: Impact of tools and feedback on judge model accuracy.