Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Off-Policy Learning with Limited Supply

Koichi Tanaka, Ren Kishimoto, Bushun Kawagishi, Yusuke Narita, Yasuo Yamamoto, Nobuyuki Shimizu, Yuta Saito

Abstract

We study off-policy learning (OPL) in contextual bandits, which plays a key role in a wide range of real-world applications such as recommendation systems and online advertising. Typical OPL in contextual bandits assumes an unconstrained environment where a policy can select the same item infinitely. However, in many practical applications, including coupon allocation and e-commerce, limited supply constrains items through budget limits on distributed coupons or inventory restrictions on products. In these settings, greedily selecting the item with the highest expected reward for the current user may lead to early depletion of that item, making it unavailable for future users who could potentially generate higher expected rewards. As a result, OPL methods that are optimal in unconstrained settings may become suboptimal in limited supply settings. To address the issue, we provide a theoretical analysis showing that conventional greedy OPL approaches may fail to maximize the policy performance, and demonstrate that policies with superior performance must exist in limited supply settings. Based on this insight, we introduce a novel method called Off-Policy learning with Limited Supply (OPLS). Rather than simply selecting the item with the highest expected reward, OPLS focuses on items with relatively higher expected rewards compared to the other users, enabling more efficient allocation of items with limited supply. Our empirical results on both synthetic and real-world datasets show that OPLS outperforms existing OPL methods in contextual bandit problems with limited supply.

Off-Policy Learning with Limited Supply

Abstract

We study off-policy learning (OPL) in contextual bandits, which plays a key role in a wide range of real-world applications such as recommendation systems and online advertising. Typical OPL in contextual bandits assumes an unconstrained environment where a policy can select the same item infinitely. However, in many practical applications, including coupon allocation and e-commerce, limited supply constrains items through budget limits on distributed coupons or inventory restrictions on products. In these settings, greedily selecting the item with the highest expected reward for the current user may lead to early depletion of that item, making it unavailable for future users who could potentially generate higher expected rewards. As a result, OPL methods that are optimal in unconstrained settings may become suboptimal in limited supply settings. To address the issue, we provide a theoretical analysis showing that conventional greedy OPL approaches may fail to maximize the policy performance, and demonstrate that policies with superior performance must exist in limited supply settings. Based on this insight, we introduce a novel method called Off-Policy learning with Limited Supply (OPLS). Rather than simply selecting the item with the highest expected reward, OPLS focuses on items with relatively higher expected rewards compared to the other users, enabling more efficient allocation of items with limited supply. Our empirical results on both synthetic and real-world datasets show that OPLS outperforms existing OPL methods in contextual bandit problems with limited supply.
Paper Structure (31 sections, 1 theorem, 30 equations, 9 figures, 2 tables)

This paper contains 31 sections, 1 theorem, 30 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Off-Policy Learning without Limited Supply
  3. Related Work
  4. Off-Policy Evaluation and Learning
  5. Recommendation with Constraints
  6. Off-Policy Learning with Limited Supply
  7. Conventional Greedy Approach
  8. The Proposed Method
  9. Extension to Mixed Supply Conditions
  10. Synthetic Experiments
  11. Setup
  12. Initial Supply
  13. Compared methods
  14. Results
  15. How does OPLS choose actions compared to the conventional greedy method?
  16. ...and 16 more sections

Key Result

Theorem 4.1

Let there be $J$ users $x_1, \ldots, x_J$ and $K$ actions $a_1, \ldots, a_K$, where each action has exactly one unit of inventory. Assume the reward function satisfies for all $j$ and $k$. Then, for any user $x_j$ and item $a_k$, the following inequality holds: where $V(\pi^*)$ denotes the value of the optimal policy and $V(\pi_{\text{greedy}})$ is the value of the greedy model-based policy.

Figures (9)

  • Figure 1: Relative policy value ($V(\pi_{\text{OPLS}})/V(\pi_{\text{greedy}})$) at each time step
  • Figure 2: The expected reward function in the experiment of Figure \ref{['fig:small-scall_timestep']}
  • Figure 3: The behavior of the conventional greedy method at $t = 10, 30, 60$
  • Figure 4: The behavior of OPLS at $t = 10, 30, 60$
  • Figure 5: Comparisons of relative policy values with varying (a)action popularity, (b)the number of users, and (c)estimation noise. The period $T$ is sufficiently large, and all items are sold out.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 4.1