VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

TL;DR

The paper addresses the cost-inefficiency of RLHF data construction by framing data collection as a monetized procurement problem and proposing VickreyFeedback, an auction-based protocol inspired by VCG to elicit truthful bidding and allocate data resources under a budget. It formalizes vanilla preferences and learning from preference datasets, introducing Direct Preference Optimization (DPO) and a cost-aware extension, QA-DPO, to weight samples by quality differences $w(b_a,b_r)=0.5+\sigma(b_a-b_r)$. The proposed protocol selects the two longest responses as winners and pays the second-longest bid, enabling cost savings while maintaining performance; experimental results show improved efficiency and competitive fine-tuning, especially on smaller datasets, with QA-DPO mitigating diversity loss. Overall, the work demonstrates a practical, scalable approach to cost-efficient RLHF data construction that can adapt to production budgets and evolving data needs.

Abstract

This paper addresses the cost-efficiency aspect of Reinforcement Learning from Human Feedback (RLHF). RLHF leverages datasets of human preferences over outputs of large language models (LLM)s to instill human expectations into LLMs. Although preference annotation comes with a monetized cost, the economic utility of a preference dataset has not been considered by far. What exacerbates this situation is that, given complex intransitive or cyclic relationships in preference datasets, existing algorithms for fine-tuning LLMs are still far from capturing comprehensive preferences. This raises severe cost-efficiency concerns in production environments, where preference data accumulate over time. In this paper, we discuss the fine-tuning of LLMs as a monetized economy and introduce an auction mechanism to improve the efficiency of preference data collection in dollar terms. We show that introducing an auction mechanism can play an essential role in enhancing the cost-efficiency of RLHF, while maintaining satisfactory model performance. Experimental results demonstrate that our proposed auction-based protocol is cost-effective for fine-tuning LLMs concentrating on high-quality feedback.

Figures (5)

• Figure 1: Illustration of VickreyFeedback for RLHF. Instruction is given to four llm agents simultaneously and each of the agents responds with a $(x, y_a, y_r)$, where $x$ represents the model's response, $y_a$ is acceptance of the response by the mechanism, $y_r$ is the model's true bid price/valuation of the respective $x$. Assuming that the quality of model responses is proportional to the length of the respective response, accepting a longer response is a proxy for selecting a higher-quality response when no additional quality control protocol is available.
• Figure 2: Results for model performance.
• Figure 3: Responses are sampled more evenly from different llms in vanilla preferences. This figure shows the distributions of responses aggregated by their source llms in the vanilla and Vickrey preferences.
• Figure 4: Vickrey preferences contain more high-quality responses. This figure shows the distributions of responses aggregated by their overall scores in the vanilla and Vickrey preferences.
• Figure 5: Analysis for model performance and data collection cost.