When Can Proxies Improve the Sample Complexity of Preference Learning?
Yuchen Zhu, Daniel Augusto de Souza, Zhengyan Shi, Mengyue Yang, Pasquale Minervini, Alexander D'Amour, Matt J. Kusner
TL;DR
This work tackles reward hacking in preference-based learning for large language models by asking when proxy preference data can provably reduce the sample complexity required to learn the true objective. It introduces a constructive, two-stage model parametrisation in which proxy components $\tilde{\phi},\tilde{\Theta},\tilde{\tau}$ are learned from abundant proxy data and an adapter $\bar{\pi}^{\dagger}$ is learned from scarce true data, yielding $\pi^{\dagger}=\tilde{\phi}\circ\tilde{\Theta}\circ\bar{\pi}^{\dagger}\circ\tilde{\tau}^{\circ}$ under several structural conditions. The key theoretical contributions prove a low-dimensional representation (via a $D$-simplex) and provide covering-number based generalisation bounds showing the second stage depends on $D$ rather than the ambient dimension $D'$, thereby achieving improved sample complexity. These results guide data collection and model design for tasks with sparse expert data, and suggest practical architectural adaptations to realize the gains in real-world LLM alignment settings. Overall, the paper offers a principled pathway to leverage proxy feedback to accelerate learning of ground-truth policies while delineating the limits of when such gains are achievable.
Abstract
We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity.