Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
Nathan Kallus
TL;DR
The paper addresses aligning LLMs to preferences when the reward–preference link is unknown, reframing the problem as semiparametric preference optimization. It shows that realizability of the divergence-constrained optimal policy in a policy class induces a semiparametric single-index model for preferences, with the index determined by the policy and the remainder as an unrestricted function. It then introduces three policy learners—PSPO, OSPO, and RSPO—along with an empirical calibration step, and proves finite-sample policy error bounds that depend on the complexity of the index class. The methods leverage first-order optimization and nuisance-function plug-ins (kernel regression, etc.) and are robust to the shape of the unknown link, enabling direct policy optimization without explicit reward modeling. This framework offers practical, scalable approaches for RLHF/RLAIF that remain valid under misspecified or unknown reward-to-preference links.
Abstract
Aligning large language models to preference data is commonly implemented by assuming a known link function between the distribution of observed preferences and the unobserved rewards (e.g., a logistic link as in Bradley-Terry). If the link is wrong, however, inferred rewards can be biased and policies be misaligned. We study policy alignment to preferences under an unknown and unrestricted link. We consider an $f$-divergence-constrained reward maximization problem and show that realizability of the solution in a policy class implies a semiparametric single-index binary choice model, where a scalar-valued index determined by a policy captures the dependence on demonstrations and the rest of the preference distribution is an unrestricted function thereof. Rather than focus on estimation of identifiable finite-dimensional structural parameters in the index as in econometrics, we focus on policy learning, focusing on error to the optimal policy and allowing unidentifiable and nonparametric indices. We develop a variety of policy learners based on profiling the link function, orthogonalizing the link function, and using link-agnostic bipartite ranking objectives. We analyze these and provide finite-sample policy error bounds that depend on generic functional complexity measures of the index class. We further consider practical implementations using first-order optimization suited to neural networks and batched data. The resulting methods are robust to unknown preference noise distribution and scale, while preserving the direct optimization of policies without explicitly fitting rewards.