Multi-Armed Bandits with Minimum Aggregated Revenue Constraints

TL;DR

This work addresses contextual multi-armed bandits with aggregated per-arm revenue guarantees, formulating the problem as a constrained planning task over context-action allocations. It introduces two algorithms, OLP and OPLP, that marry linear programming with optimistic and (for constraints) pessimistic estimates to navigate the trade-off between maximizing reward and satisfying minimum-revenue constraints. The authors prove poly-logarithmic regret alongside sublinear constraint violations for these methods, plus a lower bound showing no free exploration in the contextual setting and that the horizon dependence is tight. A refined optimality framework based on active constraint sets and a sub-optimality gap guides the learning procedure and explains how the system identifies the correct saturated constraints over time. The results offer principled strategies for context-rich environments where per-arm revenue guarantees must be respected, with implications for fair and reliable decisions in real-world sequential allocation problems.

Abstract

We examine a multi-armed bandit problem with contextual information, where the objective is to ensure that each arm receives a minimum aggregated reward across contexts while simultaneously maximizing the total cumulative reward. This framework captures a broad class of real-world applications where fair revenue allocation is critical and contextual variation is inherent. The cross-context aggregation of minimum reward constraints, while enabling better performance and easier feasibility, introduces significant technical challenges -- particularly the absence of closed-form optimal allocations typically available in standard MAB settings. We design and analyze algorithms that either optimistically prioritize performance or pessimistically enforce constraint satisfaction. For each algorithm, we derive problem-dependent upper bounds on both regret and constraint violations. Furthermore, we establish a lower bound demonstrating that the dependence on the time horizon in our results is optimal in general and revealing fundamental limitations of the free exploration principle leveraged in prior work.

Figures (4)

• Figure 1: Illustration of a MAB problem with minimum aggregated reward constraints with two arms and multiple contexts. Arm 1 is optimal in all contexts. We fix the threshold $\lambda_1$ for arm 1 and we consider how different thresholds $\lambda_{2}^{(.)}$ for arm 2 change the problem. If $\lambda_2 = \lambda_{2}^{(3)}$, the problem becomes infeasible. If $\lambda_2 = \lambda_{2}^{(2)}$, then arm 2 must be played frequently in contexts $c \geq 6$, which substantially reduces overall performance due to the large performance gap. However, if $\lambda_2 = \lambda_{2}^{(1)}$, it is sufficient to play arm 2 in contexts $c \leq 5$, where the performance gap is small, thus preserving overall reward.
• Figure 2: Display of the cumulative regret and constraint violation for OLP, OPLP and Optimistic$^3$ (denoted by OPT3), and the cumulative performance of all five algorithms, under identical conditions ($K=3$, $|\mathcal{C}|=3$, $T=50,\!000$, Gaussian distributions $\mathcal{N}(.,1)$, 5 epochs).
• Figure 3: Cumulative regret and constraint violation for OLPand OPLP, evaluated under identical conditions ($K=3$, $|\mathcal{C}|=3$,$T=500$, Gaussian distributions $\mathcal{N}(.,1)$, 5 epochs) on an instance where, according to the optimal stationary policy, no arm saturates its revenue constraint.
• Figure 4: Cumulative regret and constraint violation of OPLP under different values of the feasibility margin $\gamma^\star$. ($K=3$, $|\mathcal{C}|=3$,$T=50,\!000$, Gaussian distributions $\mathcal{N}(.,1)$, 5 epochs)

Theorems & Definitions (31)

• Definition 1
• Definition 2: Feasibility Margin
• Definition 3: Performance Sensitivity Coefficient
• Definition 4: Feasibility Gap
• Definition 5: Performance Gap
• Definition 6: Sub-optimality Gap
• Lemma 3.1: Suboptimality Characterization
• Proposition 4.0: Confidence set
• Theorem 5.1: Upper bounds for OLP
• Theorem 5.2: Upper bounds for OPLP