Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration

TL;DR

This work tackles the opacity of Bayesian optimization by introducing ShapleyBO, which uses Shapley values to attribute a BO proposal's informativeness to individual parameters. By decomposing the acquisition function into mean (exploitation) and uncertainty (exploration) components, and further separating epistemic from aleatoric uncertainty, ShapleyBO provides a principled, online explanation of why BO selects particular configurations. The authors integrate these explanations into a human-machine interface to enable intervention when proposals misalign with human reasoning, and demonstrate the approach on an exosuit personalization task with human-in-the-loop optimization, achieving faster convergence and lower regret in several subjects. The work highlights the practical impact of interpretable optimization in human-AI collaboration, with potential extensions to multi-criteria BO and theoretical guarantees for Shapley-guided interventions.

Abstract

Bayesian optimization (BO) with Gaussian processes (GP) has become an indispensable algorithm for black box optimization problems. Not without a dash of irony, BO is often considered a black box itself, lacking ways to provide reasons as to why certain parameters are proposed to be evaluated. This is particularly relevant in human-in-the-loop applications of BO, such as in robotics. We address this issue by proposing ShapleyBO, a framework for interpreting BO's proposals by game-theoretic Shapley values.They quantify each parameter's contribution to BO's acquisition function. Exploiting the linearity of Shapley values, we are further able to identify how strongly each parameter drives BO's exploration and exploitation for additive acquisition functions like the confidence bound. We also show that ShapleyBO can disentangle the contributions to exploration into those that explore aleatoric and epistemic uncertainty. Moreover, our method gives rise to a ShapleyBO-assisted human machine interface (HMI), allowing users to interfere with BO in case proposals do not align with human reasoning. We demonstrate this HMI's benefits for the use case of personalizing wearable robotic devices (assistive back exosuits) by human-in-the-loop BO. Results suggest human-BO teams with access to ShapleyBO can achieve lower regret than teams without.

Figures (24)

• Figure 1: ShapleyBO results in iteration 59 of BO on $\Psi(\bm \theta) = f(\bm \theta) + \bm \epsilon$. Plots i and ii for $\lambda = 1$ and plots iii and iv for $\lambda = 10$, see croppi2021explaining. Contributions ($phi$) are averaged over 30 proposals for each $\lambda$. On the right, the overall informativeness of the parameters is displayed ($cb$ contributions), and on the left the decomposition into $m$ (red) and $se$ (blue) contributions. Recall that cb is minimized. Error bars: Standard deviation of estimates.
• Figure 2: Informativeness paths for hyper ellipsoid optimization, see croppi2021explaining. Plot on top displays $cb$ contributions with facets for parameters (vertical) and $\lambda$ (horizontal); beneath its decomposition into $m$ (red) and $se$ (blue) contributions. They are averaged over 30 proposals in each iteration and the uncertainty of the estimate is displayed with error bars using one standard deviation. The black dot-dashed line in the $\lambda = 10$ plots displays the average contribution of the parameters in the $\lambda = 1$ run.
• Figure 3: Contour plots of noisy ellipsoid function $\bm \Psi(\bm \theta) = g(\bm \theta) + \bm \epsilon(\bm \theta)$, see Equations \ref{['eq:ellipsoid']} and \ref{['eq:ellipsoid-noise']}. Red: low $\bm \Psi(\bm \theta)$; blue: high $\bm \Psi(\bm \theta)$.
• Figure 4: A: Assistive soft back exosuit. B: Force profile example for preference learning. Subjects are asked to compare controllers setting 1 (pink) to 2 (blue). Each option varies in the amount of lowering gain ($\theta_{low}$) and lifting gain ($\theta_{lif}$), see Arens2023Preference.
• Figure 5: Results of Agents A0-A4 (see Table \ref{['tab:my_label']}) in human-AI collaborative BO for simulated exosuit personalization (individual 1) with 10 iterations and 3 initial samples each. Error bars indicate $95 \%$ confidence intervals; $k=2$ for A3, $\beta = 2$ for A2 and A4.