Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Episodic Contextual Bandits with Knapsacks under Conversion Models

Wang Chi Cheung, Zitian Li

TL;DR

This paper extends contextual bandits-with-knapsacks to episodic settings with non-stationary context distributions and a shared latent conversion model across episodes. It introduces Mimic-Opt-DP, an online algorithm that combines backward optimistic DP with a confidence-bound oracle to handle unknown rho and time-varying Lambda_h, achieving Reg(T) = o(T) without dependence on the size of the context space. The method uses a careful data-splitting scheme between labeled data for rho and unlabeled data for learning context distributions, with a lazy update rule to facilitate analysis; unlabeled feature data can further improve regret in several regimes. The framework unifies several CB models (GLM, logistic, GP, kernelized) via the CB oracle and yields concrete regret bounds for applications like dynamic pricing and first-price auctions, while highlighting the benefits and limitations of single vs multiple resource constraints.

Abstract

We study an online setting, where a decision maker (DM) interacts with contextual bandit-with-knapsack (BwK) instances in repeated episodes. These episodes start with different resource amounts, and the contexts' probability distributions are non-stationary in an episode. All episodes share the same latent conversion model, which governs the random outcome contingent upon a request's context and an allocation decision. Our model captures applications such as dynamic pricing on perishable resources with episodic replenishment, and first price auctions in repeated episodes with different starting budgets. We design an online algorithm that achieves a regret sub-linear in $T$, the number of episodes, assuming access to a \emph{confidence bound oracle} that achieves an $o(T)$-regret. Such an oracle is readily available from existing contextual bandit literature. We overcome the technical challenge with arbitrarily many possible contexts, which leads to a reinforcement learning problem with an unbounded state space. Our framework provides improved regret bounds in certain settings when the DM is provided with unlabeled feature data, which is novel to the contextual BwK literature.

Episodic Contextual Bandits with Knapsacks under Conversion Models

TL;DR

This paper extends contextual bandits-with-knapsacks to episodic settings with non-stationary context distributions and a shared latent conversion model across episodes. It introduces Mimic-Opt-DP, an online algorithm that combines backward optimistic DP with a confidence-bound oracle to handle unknown rho and time-varying Lambda_h, achieving Reg(T) = o(T) without dependence on the size of the context space. The method uses a careful data-splitting scheme between labeled data for rho and unlabeled data for learning context distributions, with a lazy update rule to facilitate analysis; unlabeled feature data can further improve regret in several regimes. The framework unifies several CB models (GLM, logistic, GP, kernelized) via the CB oracle and yields concrete regret bounds for applications like dynamic pricing and first-price auctions, while highlighting the benefits and limitations of single vs multiple resource constraints.

Abstract

We study an online setting, where a decision maker (DM) interacts with contextual bandit-with-knapsack (BwK) instances in repeated episodes. These episodes start with different resource amounts, and the contexts' probability distributions are non-stationary in an episode. All episodes share the same latent conversion model, which governs the random outcome contingent upon a request's context and an allocation decision. Our model captures applications such as dynamic pricing on perishable resources with episodic replenishment, and first price auctions in repeated episodes with different starting budgets. We design an online algorithm that achieves a regret sub-linear in , the number of episodes, assuming access to a \emph{confidence bound oracle} that achieves an -regret. Such an oracle is readily available from existing contextual bandit literature. We overcome the technical challenge with arbitrarily many possible contexts, which leads to a reinforcement learning problem with an unbounded state space. Our framework provides improved regret bounds in certain settings when the DM is provided with unlabeled feature data, which is novel to the contextual BwK literature.

Paper Structure

This paper contains 28 sections, 8 theorems, 107 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Model
  4. Characterizing $\textsf{opt}_1\ldots, \textsf{opt}_T$ with Bellman Equations
  5. Online Algorithms
  6. Algorithm Design
  7. Main Results
  8. A High Level View on the Proof of Theorem \ref{['thm:main']}
  9. Benefits of Unlabeled Feature Data
  10. Supplemental Discussions
  11. Discussions on the Model Applications
  12. Dynamic Pricing
  13. Dynamic Procurement
  14. First Price Auction with Finite Budgets in Repeated Episodes
  15. Examples of Confidence Bound (CB) Oracles
  16. ...and 13 more sections

Key Result

Theorem 3.1

Consider Mimic-Opt-DP with CB oracle ${\cal O}, \delta\in (0, 1)$, $\{{\cal J}_t\}^\infty_{t=1}$ specified in (eq:J) and ${\cal S}_1 = \emptyset$. With probability at least $1-9\delta$, it holds that $\text{Reg}(T) \leq$ where $\tilde{O}(\cdot)$ hides a multiplicative factor of absolute constant times $\sqrt{\log(LHT/\delta)}$, and $O(\cdot)$ hides a multiplicative factor of absolute constant. In

Figures (1)

  • Figure 1: Numeric Performance of Micmic-Opt-DP and Benchmark Algorithms

Theorems & Definitions (8)

  • Theorem 3.1
  • Lemma 3.2
  • Corollary 3.3
  • Lemma 3.4
  • Theorem 3.5
  • Lemma 3.6
  • Corollary 3.7
  • Proposition B.1: Azuma-Hoeffding Inequality