UCB-type Algorithm for Budget-Constrained Expert Learning

TL;DR

This work addresses budget-constrained learning with multiple self-learning experts by introducing the M-LCB meta-algorithm, which governs a pool of $K$ adaptively learning predictors under a per-round budget $M$. The algorithm builds confidence bounds directly from realized losses, enabling anytime regret guarantees in stochastic settings, and extends naturally to multi-play scenarios. When each expert has an internal regret of $\tilde O(T^{\alpha})$, M-LCB achieves a global regret of $\tilde O\left(\sqrt{\tfrac{KT}{M}} + (\tfrac{K}{M})^{1-\alpha} T^{\alpha}\right)$, and lower bounds establish near-optimality up to logarithmic factors. The framework covers parametric online models and bandit-based experts, illustrating a principled extension of classical bandit theory to coordinating stateful, self-learning experts under limited resources, with potential for contextual and broader resource-aware applications.

Abstract

In many modern applications, a system must dynamically choose between several adaptive learning algorithms that are trained online. Examples include model selection in streaming environments, switching between trading strategies in finance, and orchestrating multiple contextual bandit or reinforcement learning agents. At each round, a learner must select one predictor among $K$ adaptive experts to make a prediction, while being able to update at most $M \le K$ of them under a fixed training budget. We address this problem in the \emph{stochastic setting} and introduce \algname{M-LCB}, a computationally efficient UCB-style meta-algorithm that provides \emph{anytime regret guarantees}. Its confidence intervals are built directly from realized losses, require no additional optimization, and seamlessly reflect the convergence properties of the underlying experts. If each expert achieves internal regret $\tilde O(T^α)$, then \algname{M-LCB} ensures overall regret bounded by $\tilde O\!\Bigl(\sqrt{\tfrac{KT}{M}} \;+\; (K/M)^{1-α}\,T^α\Bigr)$. To our knowledge, this is the first result establishing regret guarantees when multiple adaptive experts are trained simultaneously under per-round budget constraints. We illustrate the framework with two representative cases: (i) parametric models trained online with stochastic losses, and (ii) experts that are themselves multi-armed bandit algorithms. These examples highlight how \algname{M-LCB} extends the classical bandit paradigm to the more realistic scenario of coordinating stateful, self-learning experts under limited resources.

Figures (2)

• Figure 1: Nonlinear link functions associated with the arms (top) and the density of generated data points (bottom). One can see that the last three functions are highly similar where the data is concentrated, making it hard to distinguish the optimal arm.
• Figure 2: Performance comparison on the GLM model selection problem. (a) Cumulative regret. (b) Final distribution of arm selection. (c) Allocation of computational budget across arms.

Theorems & Definitions (39)

• Remark 1: On the bounded loss
• Remark 2: Example
• Remark 3: On $(U_k,\delta)$-bounds
• Definition 1: UCB and LCB
• Definition 2: Stochastic tasks with $\alpha$–regret lower bound
• Theorem 1: Lower bound
• Lemma 1: Anytime confidence bounds
• Lemma 2: Regret bounds
• Theorem 2: Convergence rates
• Lemma 3: Top-$M$ experts regret