Knowledge Gradient for Preference Learning

TL;DR

This work tackles learning from pairwise preferences in Bayesian optimization by deriving an exact analytic knowledge gradient for preferential Bayesian optimization under a Gaussian process prior and probit likelihood. The key insight is that the one-step look-ahead posterior conditioned on a dueling outcome is an extended skew normal distribution, enabling a closed-form expression for the look-ahead mean and thus an exact KG without resorting to common approximations. Empirically, the exact KG demonstrates competitive performance across multiple benchmarks and noise regimes, with a case study highlighting how KG’s behavior can differ from EUBO in practice. The results extend the applicability of KG to preference learning and open doors to future work on skew GP models and broader look-ahead acquisition schemes.

Abstract

The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.

Figures (4)

• Figure 1: A one-dimensional example of preferential knowledge gradient. The ground-truth latent function is a quadratic function $f(x) = -x^2$, whose maximum is attained at the origin. Left & Mid: The GP model fit before and after the query. The blue shaded region indicates two standard deviations of the GP posterior. The green circles and red crosses are the observed preferential comparisons. The arrows point from duel winners to duel losers. The two triangles indicate the query candidates selected by maximizing the knowledge gradient. Right: The landscape of knowledge gradient $\operatorname{kg}(x_1, x_2)$ by maxing out the fantasy samples $\mathbf{x}_{\pm}$.
• Figure 2: (Low Noise) The optimality gaps in log scales against the number of queries for random sampling, log expected improvement, EUBO, and the knowledge gradient acquisition function. A small amount of noise is added to the pairwise comparisons to simulate noisy preferences. The knowledge gradient acquisition function outperforms all baselines on all problems except one. Note that the Ackley function here uses the original domain $[-32.768, 32.768]^d$.
• Figure 3: (High Noise) The optimality gaps in log scales against the number of queries for random sampling, log expected improvement, EUBO, and the knowledge gradient acquisition function. The setting is the same as in \ref{['fig:benchmark-noisy-low']} except that the pairwise comparisons are noisier.
• Figure 4: A comparison of the queries selected by EUBO and the knowledge gradient on the 2D Levy function. The preferential outcomes are generated deterministically in this case study, i.e., the noise parameter $\sigma$ in the preferetial probit likelihood \ref{['eq:preferential-probit-likelihood']} vanishes. Left: The queries selected by Sobol during initialization and the ground-truth latent utility function. Mid: The queries selected by EUBO after $50$ queries and the GP posterior mean. Right: The queries selected by the knowledge gradient after $50$ queries and the GP posterior mean. Even though both methods identify the region of the optimizer, the queries have very different patterns. In particular, EUBO tends to "collapse" the two queries close to the estimated maximum, while the knowledge gradient selects queries centers around the estimated maximum. In this case, the optimality gap of EUBO is $0.0182$ while that of the knowledge gradient is $0.0191$ and EUBO achieves a better performance.

Theorems & Definitions (6)

• Lemma 1
• proof
• Definition 1: Standard Form
• Definition 2: General Form
• Lemma 2
• proof