Offline Dynamic Inventory and Pricing Strategy: Addressing Censored and Dependent Demand

TL;DR

This paper tackles offline joint pricing and inventory control under censored and dependent demand by modeling the environment as a high-order Markov decision process and introducing two data-driven algorithms, C-FQI and PC-FQI. It combines offline reinforcement learning with survival analysis, imputing censored rewards via Kaplan-Meier estimates and solving high-order Bellman equations to learn near-optimal policies. A key concept, censoring coverage, quantifies offline data adequacy for recovering the optimal policy, and finite-sample regret bounds validate the methods under general conditions. Numerical experiments show the framework can approximate optimal policies as the offline data size grows, while quantifying the cost of demand censoring and the benefits of pessimistic learning in uncertain data regimes.

Abstract

In this paper, we study the offline sequential feature-based pricing and inventory control problem where the current demand depends on the past demand levels and any demand exceeding the available inventory is lost. Our goal is to leverage the offline dataset, consisting of past prices, ordering quantities, inventory levels, covariates, and censored sales levels, to estimate the optimal pricing and inventory control policy that maximizes long-term profit. While the underlying dynamic without censoring can be modeled by Markov decision process (MDP), the primary obstacle arises from the observed process where demand censoring is present, resulting in missing profit information, the failure of the Markov property, and a non-stationary optimal policy. To overcome these challenges, we first approximate the optimal policy by solving a high-order MDP characterized by the number of consecutive censoring instances, which ultimately boils down to solving a specialized Bellman equation tailored for this problem. Inspired by offline reinforcement learning and survival analysis, we propose two novel data-driven algorithms to solving these Bellman equations and, thus, estimate the optimal policy. Furthermore, we establish finite sample regret bounds to validate the effectiveness of these algorithms. Finally, we conduct numerical experiments to demonstrate the efficacy of our algorithms in estimating the optimal policy. To the best of our knowledge, this is the first data-driven approach to learning optimal pricing and inventory control policies in a sequential decision-making environment characterized by censored and dependent demand. The implementations of the proposed algorithms are available at https://github.com/gundemkorel/Inventory_Pricing_Control

Figures (6)

• Figure 1: Transition dynamics of the underlying decision process ${\{S_t, A_t, R_t\}}_{t \geq 0}$; Given the current state $S_t$ and action $A_t$, the transition to the next state $S_{t+1}$ and reward $R_t$ is independent of all prior states and actions.
• Figure 2: Depiction of a trajectory under the observed process $\{W_t, A_t, R_t \mathbb{I}[\Delta_t = 1]\}_{t \geq 0}$, where $\Delta_t$ is the censoring indicator. When $\Delta_t = 1$ (no censoring), the reward, $R_t$, and the demand, $D_t$, are observable, rendering $S_{t+1}=W_{t+1}\setminus\Delta_{t}=1$. However, when censoring occurs while moving from time step $t+1$ to $t+2$ (i.e $\Delta_{t+1} = 0$) , the reward $R_{t+1}$ and demand $D_{t+1}$ become unobservable, and thus, the state $S_{t+2}$. This implies that $S_{t+2}\neq W_{t+2}\setminus\Delta_{t+1}=0$. The shaded boxes represent these censored elements, highlighting the transition to limited observability under censoring, where $W_{t+2}$ does not contain as much information as $S_{t+2}$. This figure emphasizes the impact of censoring on the process dynamics.
• Figure 3: Simplified illustration of offline data with respect to consecutive censoring. The x-axis represents the number of consecutive censoring instances, while the y-axis (A) shows the action space. Green dots represent observed state-action pairs in the offline data. For example, there are two observations with 4 consecutive censoring instances where actions $a_2$ and $a_4$ were taken. The vertical dashed blue line indicates the maximum consecutive censoring in the offline data. The red shaded area with a question mark represents the region where the learned policy lacks training data, potentially leading to suboptimal decisions when deployed in the real world if encountered.
• Figure 4: Illustrations of Censoring Coverage between Optimal and Behavior Policies. (a) Sufficient censoring coverage scenario: the behavior policy (blue) has sufficient censoring coverage over the optimal policy (red), and thus, all the trajectories under the optimal policy can be learnt from the offline data as indicated by green shaded area. (b) Insufficient censoring coverage scenario: the behavior policy does not have sufficient censoring coverage over the the optimal policy, and thus, the trajectories under the optimal policy with more than $n_{T',b}$ consecutive censoring can not be learnt from the offline data as illustrated by red shaded area. The x-axis represents the window size, while the y-axis shows the number of consecutive censoring instances. $n_{T',b}$ and $n_{T',*}$ represent the maximum number of consecutive censoring instances for the behavior and optimal policies within the window size of $T'$, respectively.
• Figure 5: Illustration of learnable and unlearnable trajectories of $\pi^*$ based on consecutive censoring patterns, assuming $n_{K,b}=2$. Green circles indicate uncensored observations, while red circles represent censored observations. In the top panel, two learnable trajectories are shown, where the number of consecutive censored instances (red) within the first $K$ horizon does not exceed $n_{K,b}$, set to 2 in this example. The bottom panel presents two unlearnable trajectories, where the number of consecutive censored instances within the first $K$ horizon exceeds $n_{K,b}$. This visualization highlights how the censoring pattern within the initial $K$ horizon influences whether a trajectory is learnable.

Theorems & Definitions (36)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Lemma 1
• Lemma 2
• Lemma 3
• Definition 7: $\epsilon$-Uncertainty Quantifier