Budget Allocation for Unknown Value Functions in a Lipschitz Space

TL;DR

Budget Allocation for Unknown Value Functions in a Lipschitz Space introduces Unknown Value Probing (UVP), a framework for allocating a fixed budget to identify the best model configuration when the value functions are unknown but Lipschitz-continuous. It develops clustering-based algorithms—FullCent, Enhanced-FullCent, AdaCent, and Enhanced-AdaCent—with provable approximation guarantees around $(1 - 2\epsilon r_k^*)$, supported by hardness results. The methods incorporate enhanced distance measures and adaptive pruning to focus exploration on high-potential regions, achieving strong empirical performance on diverse hyperparameter optimization benchmarks (e.g., YAHPO Gym, lcbench, rbv2). The findings show that exploiting the problem's structure yields efficient budget usage and improved early stopping, offering practical impact for budgeted model selection in ML pipelines.

Abstract

Building learning models frequently requires evaluating numerous intermediate models. Examples include models considered during feature selection, model structure search, and parameter tunings. The evaluation of an intermediate model influences subsequent model exploration decisions. Although prior knowledge can provide initial quality estimates, true performance is only revealed after evaluation. In this work, we address the challenge of optimally allocating a bounded budget to explore the space of intermediate models. We formalize this as a general budget allocation problem over unknown-value functions within a Lipschitz space.

Figures (15)

• Figure 1: Illustrative example where adjusted distances yield better clustering, resulting in probing the highest value point.
• Figure 2: Mean rank (averaged over all datasets) of each algorithm over 30 runs on the three YAHPO scenarios. Lower rank for an algorithm indicates a higher validation accuracy for the same budget. The $x$-axis denotes fraction of total budget (epochs for lcbench, data fraction for rbv2_rpart and rbv2_aknn), starting at 0.1 to emphasize early-phase differences.
• Figure 3: Validation accuracy curves on 9 datasets from lcbench. Both AdaCent and Enhanced-AdaCentEnhanced-AdaCent consistently converge faster and achieve higher accuracy. Similar trends are observed across 35 datasets, as shown in Appendix \ref{['sec:figs']}.
• Figure 4: Validation accuracy curves for 6 datasets from rbv2_rpart. Both AdaCent and Enhanced-AdaCentEnhanced-AdaCent consistently outperform the baselines. Similar trends are observed across 107 datasets, as shown in Appendix \ref{['sec:figs']}.
• Figure 5: Validation accuracy curves for 6 datasets from rbv2_aknn. Both AdaCent and Enhanced-AdaCentEnhanced-AdaCent consistently outperform the baselines. Similar trends are observed across 118 datasets, as shown in Appendix \ref{['sec:figs']}.

Theorems & Definitions (37)

• Theorem 1.3: See Theorem \ref{['thm:FC-approx']} and Corollary \ref{['col:FC-hard']}
• Theorem 1.4: See Theorem \ref{['thm:EFC-approx']}
• Theorem 1.5: See Theorem \ref{['thm:AC-approx']} and Corollary \ref{['col:AC-hard']}
• Lemma 3.1
• proof : Proof of Lemma \ref{['lemma:FC-budget']}
• Lemma 3.2
• proof : Proof of Lemma \ref{['lemma:FC-comp']}
• Theorem 3.3
• proof : Proof of Theorem \ref{['thm:FC-approx']}
• Theorem 3.4