Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints

Udvas Das, Debabrota Basu

TL;DR

The paper addresses pure exploration in multi-armed bandits under unknown linear constraints, aiming to identify an $r$-optimal feasible policy with high confidence. It introduces a Lagrangian relaxation of the lower bound and develops two algorithms, LATS and LAGEX, that jointly estimate rewards and constraints while tracking the lower bound through optimistic feasible sets and a constraint-adaptive stopping rule. Theoretical results establish strong duality, convergence of constraint estimates, and asymptotic optimality: LAGEX achieves the asymptotically optimal sample complexity, while LATS matches it up to constraint-dependent constants via a shadow-price measure. Empirical evaluations on synthetic and real data confirm that LAGEX often has the best efficiency and safety properties, validating the approach and highlighting how constraint geometry impacts hardness and performance.

Abstract

Pure exploration in bandits formalises multiple real-world problems, such as tuning hyper-parameters or conducting user studies to test a set of items, where different safety, resource, and fairness constraints on the decision space naturally appear. We study these problems as pure exploration in multi-armed bandits with unknown linear constraints, where the aim is to identify an $r$-optimal and feasible policy as fast as possible with a given level of confidence. First, we propose a Lagrangian relaxation of the sample complexity lower bound for pure exploration under constraints. Second, we leverage properties of convex optimisation in the Lagrangian lower bound to propose two computationally efficient extensions of Track-and-Stop and Gamified Explorer, namely LATS and LAGEX. Then, we propose a constraint-adaptive stopping rule, and while tracking the lower bound, use optimistic estimate of the feasible set at each step. We show that LAGEX achieves asymptotically optimal sample complexity upper bound, while LATS shows asymptotic optimality up to novel constraint-dependent constants. Finally, we conduct numerical experiments with different reward distributions and constraints that validate efficient performance of LATS and LAGEX.

Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints

TL;DR

The paper addresses pure exploration in multi-armed bandits under unknown linear constraints, aiming to identify an -optimal feasible policy with high confidence. It introduces a Lagrangian relaxation of the lower bound and develops two algorithms, LATS and LAGEX, that jointly estimate rewards and constraints while tracking the lower bound through optimistic feasible sets and a constraint-adaptive stopping rule. Theoretical results establish strong duality, convergence of constraint estimates, and asymptotic optimality: LAGEX achieves the asymptotically optimal sample complexity, while LATS matches it up to constraint-dependent constants via a shadow-price measure. Empirical evaluations on synthetic and real data confirm that LAGEX often has the best efficiency and safety properties, validating the approach and highlighting how constraint geometry impacts hardness and performance.

Abstract

Pure exploration in bandits formalises multiple real-world problems, such as tuning hyper-parameters or conducting user studies to test a set of items, where different safety, resource, and fairness constraints on the decision space naturally appear. We study these problems as pure exploration in multi-armed bandits with unknown linear constraints, where the aim is to identify an -optimal and feasible policy as fast as possible with a given level of confidence. First, we propose a Lagrangian relaxation of the sample complexity lower bound for pure exploration under constraints. Second, we leverage properties of convex optimisation in the Lagrangian lower bound to propose two computationally efficient extensions of Track-and-Stop and Gamified Explorer, namely LATS and LAGEX. Then, we propose a constraint-adaptive stopping rule, and while tracking the lower bound, use optimistic estimate of the feasible set at each step. We show that LAGEX achieves asymptotically optimal sample complexity upper bound, while LATS shows asymptotic optimality up to novel constraint-dependent constants. Finally, we conduct numerical experiments with different reward distributions and constraints that validate efficient performance of LATS and LAGEX.

Paper Structure

This paper contains 34 sections, 24 theorems, 119 equations, 12 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Exploration under Unknown Constraints
  3. Extension of Prior Bandit Problems
  4. Lagrangian Relaxation of the Lower Bound
  5. Information Acquisition: Estimate Constraints
  6. Relaxation with Estimated Constraints
  7. Algorithm Design
  8. LATS.
  9. Experimental Analysis
  10. Discussion & Future Work
  11. Notations
  12. Discussion on The Problem Setting and Connection to Literature
  13. Extended Related Work
  14. Motivations: Reductions to and Generalisations of The Existing BAI Settings
  15. Strong Duality and the Lagrangian Multiplier: Proof of Theorem \ref{['thm:Existence of strong duality and bound on Lagrangian multiplier']}
  16. ...and 19 more sections

Key Result

Lemma 1

At any time $t \in \mathbb{N}$, we construct the optimistic feasible policy set such that with probability $1-\delta$, $\Hat{\mathcal{F}}_t \mathrel{\triangleq} \{\boldsymbol{\pi} \in \Delta_{K}: \min_{A' \in \mathcal{C}_t} A'\boldsymbol{\pi} \leq \boldsymbol{0}\}$, satisfies $\mathcal{F} \subseteq

Figures (12)

  • Figure 1: Effect of rank-one update of active constraint
  • Figure 2: Convergence of the optimistic feasible set and optimal policy.
  • Figure 3: Normal cones over time.
  • Figure 4: Lower bounds with and without constraints, and LAGEX for $\mu_3 \in[0.5,2.5]$ in Setup 1.
  • Figure 5: Sample complexity (median$\pm$std.) of algorithms for hard env. in Setup 1.
  • ...and 7 more figures

Theorems & Definitions (42)

  • Definition 1: $(1-\delta)$-correct and $(1-\delta)$-feasible $r$-optimal pure exploration
  • Lemma 1: Optimistic feasible sets
  • Remark 1
  • Theorem 1: Strong Duality and Range of Lagrange Multipliers
  • Remark 2: Connections with Lagrangian-based Methods in Bandits.
  • Proposition 1: Projection Lemma for Unknown Constraints
  • Theorem 2
  • Theorem 3: Existence of unique optimal allocation
  • Theorem 4
  • Corollary 1
  • ...and 32 more