Pareto-Optimal Anytime Algorithms via Bayesian Racing

TL;DR

PolarBear is introduced, a procedure that identifies the anytime Pareto set through adaptive sampling with calibrated uncertainty, and Bayesian inference over a temporal Plackett-Luce ranking model provides posterior beliefs about pairwise dominance, enabling early elimination of confidently dominated algorithms.

Abstract

Selecting an optimization algorithm requires comparing candidates across problem instances, but the computational budget for deployment is often unknown at benchmarking time. Current methods either collapse anytime performance into a scalar, require manual interpretation of plots, or produce conclusions that change when algorithms are added or removed. Moreover, methods based on raw objective values require normalization, which needs bounds or optima that are often unavailable and breaks coherent aggregation across instances. We propose a framework that formulates anytime algorithm comparison as Pareto optimization over time: an algorithm is non-dominated if no competitor beats it at every timepoint. By using rankings rather than objective values, our approach requires no bounds, no normalization, and aggregates coherently across arbitrary instance distributions without requiring known optima. We introduce PolarBear (Pareto-optimal anytime algorithms via Bayesian racing), a procedure that identifies the anytime Pareto set through adaptive sampling with calibrated uncertainty. Bayesian inference over a temporal Plackett-Luce ranking model provides posterior beliefs about pairwise dominance, enabling early elimination of confidently dominated algorithms. The output Pareto set together with the posterior supports downstream algorithm selection under arbitrary time preferences and risk profiles without additional experiments.

Figures (13)

• Figure 1: Mean execution time in seconds (5 repetitions) for selected models and approximation methods.
• Figure 2: A synthetic example with 5 algorithms. Figures (b) to (e) show the mean and 95% credible interval of the rating posterior (y-axis) over time (x-axis), with ground truth ratings from (a) as dotted lines. Figure (f) shows the resulting posterior over preference values for a uniform budget preference. Note that ratings at a specific $t$ always sum to 1.
• Figure 3: Dominance posterior probabilities over rounds. Points show $\max_{A \in \mathbf{A}} P(A \overset{\mathbf{T}}{\succ} B)$ for each algorithm $B$ at each round (left y-axis). Note that the left y-axis changes to a log-scale for $p>0.9$. A cross marks elimination, i.e., when the dominance probability climbs above the decision threshold $\alpha$. A diamond marks that the algorithm is fully resolved and a Pareto set member. The black line shows the proportion of resolved algorithm pairs (right y-axis).
• Figure 4: A synthetic example with 10 algorithms. Conventions as in \ref{['fig:synthetic-A']}.
• Figure 5: A synthetic example with 10 algorithms. Conventions as in \ref{['fig:synthetic-A']}. A resolved algorithm has all pairwise relationships determined; for non-eliminated algorithms, this confirms Pareto set membership.

Theorems & Definitions (1)

• proposition 1