Heteroscedastic Preferential Bayesian Optimization with Informative Noise Distributions

TL;DR

This work addresses preferential Bayesian optimization (PBO) by modeling human uncertainty as input-dependent (heteroscedastic) noise rather than a fixed variance. It introduces an anchor-based KDE noise model that assigns lower noise near reliable inputs and higher noise elsewhere, with the variance σ^2_ε(x) set to a exp(- hat p(x|h, X0)). Inference uses Hallucination Believer to handle the skew GP posterior, and acquisition functions are made risk-averse via ANPEI and RAHBO to balance informativeness with human ease of comparison. Theoretical guarantees show convergence rates for the variance estimator under Hölder smoothness and concentration bounds, while experiments on synthetic benchmarks demonstrate improved mean-variance objectives and more stable, low-noise querying compared to homoscedastic baselines. The approach offers a principled framework for incorporating human partial knowledge into PBO, with potential impact on efficient design optimization across domains requiring human-in-the-loop feedback.

Abstract

Preferential Bayesian optimization (PBO) is a sample-efficient framework for learning human preferences between candidate designs. PBO classically relies on homoscedastic noise models to represent human aleatoric uncertainty. Yet, such noise fails to accurately capture the varying levels of human aleatoric uncertainty, particularly when the user possesses partial knowledge among different pairs of candidates. For instance, a chemist with solid expertise in glucose-related molecules may easily compare two compounds from that family while struggling to compare alcohol-related molecules. Currently, PBO overlooks this uncertainty during the search for a new candidate through the maximization of the acquisition function, consequently underestimating the risk associated with human uncertainty. To address this issue, we propose a heteroscedastic noise model to capture human aleatoric uncertainty. This model adaptively assigns noise levels based on the distance of a specific input to a predefined set of reliable inputs known as anchors provided by the human. Anchors encapsulate partial knowledge and offer insight into the comparative difficulty of evaluating different candidate pairs. Such a model can be seamlessly integrated into the acquisition function, thus leading to candidate design pairs that elegantly trade informativeness and ease of comparison for the human expert. We perform an extensive empirical evaluation of the proposed approach, demonstrating a consistent improvement over homoscedastic PBO.

Figures (8)

• Figure 1: Heteroscedastic Preferential Bayesian Optimization. Top left: latent user utility with heteroscedastic noisy evaluations plotted as an example. Top right: ground truth latent user uncertainty used for this example. Middle: Preferential GP surrogates obtained using queries from vanilla AF (left) and user-uncertainty-aware AF (right). Our anchor-based model of user uncertainty leads to queries associated with lower noise and yet similarly high values, resulting in a better-calibrated surrogate (blue) compared to the vanilla GP (red, left). Bottom: AF landscape. The estimated user uncertainty (green) informs a user-uncertainty-aware acquisition function (blue), leading to a maximizer that differs from the vanilla AF (red) and accurately selects the low-variance design.
• Figure 2: Results for three synthetic problems: Sine1D (a), Branin2D (b) and Hartmann4D (c). (a.1) Simple regret. (a.2) Risk-averse simple regret. (a.3) Cumulative regret. (a.4) Risk-averse cumulative regret. (b.1) Best value found. (b.2) Risk-averse best value found (b.3) Inferred noise variance $\hat{\sigma}_{\varepsilon}({\bf{x}})$. (c.1) Best value found. (c.2) Risk-averse best value found (c.3) Inferred noise variance $\hat{\sigma}_{\varepsilon}({\bf{x}})$. For all examples, $\rho = 3 f({\bf{x}}_{\max})$. Mean and standard deviations were computed across 30 random seeds. Risk-averse AFs outperform vanilla AFs, specifically for cumulative regret metrics.
• Figure 3: Results for the Hartmann4D test function using different approximated inference techniques.
• Figure S1: Additional visualizations of query acquisition in the sine experiments. Each column corresponds to a certain initialization. For the same initialization, PBO-ANPEI tends to exploit regions with low noise, whereas PBO-EI's queries are concentrated in high-variance regions.
• Figure S2: Additional visualizations of query acquisition in the sine experiments. Each column corresponds to a certain initialization. For the same initialization, PBO-RAHBO tends to exploit regions with low noise, whereas PBO-UCB's queries are concentrated in high-variance regions.

Theorems & Definitions (15)

• Theorem 4.2
• Theorem 4.3
• Proposition A.1
• Definition C.1
• Definition C.2
• Definition C.3
• Proposition C.4
• Proposition C.5
• Lemma C.6
• Theorem C.6