Choice-Model-Assisted Q-learning for Delayed-Feedback Revenue Management

TL;DR

This work addresses delayed-feedback revenue management by integrating a fixed discrete choice model (DCM) as a partial world model into Q-learning. The proposed Choice-Model-Assisted DQN (CA-DQN) imputes delayed rewards using the offline-calibrated DCM, enabling immediate learning updates and a two-timescale interpretation of model-learning with value-learning. Theoretical analysis shows that with a fixed DCM, Q-learning converges to a neighborhood of the true Q-function with a bound that scales as $O(\varepsilon/(1-\gamma) + t^{-1/2})$, where $\varepsilon$ captures partial-model error; the neighborhood tightens as more data accrues. Empirical validation on a simulator calibrated to 61,619 hotel bookings demonstrates no difference under stationarity, potential gains under in-family shifts (up to 12.4%), and robust degradation under structural misspecification, highlighting when partial behavioral models improve robustness and when they bias outcomes.

Abstract

We study reinforcement learning for revenue management with delayed feedback, where a substantial fraction of value is determined by customer cancellations and modifications observed days after booking. We propose \emph{choice-model-assisted RL}: a calibrated discrete choice model is used as a fixed partial world model to impute the delayed component of the learning target at decision time. In the fixed-model deployment regime, we prove that tabular Q-learning with model-imputed targets converges to an $O(\varepsilon/(1-γ))$ neighborhood of the optimal Q-function, where $\varepsilon$ summarizes partial-model error, with an additional $O(t^{-1/2})$ sampling term. Experiments in a simulator calibrated from 61{,}619 hotel bookings (1{,}088 independent runs) show: (i) no statistically detectable difference from a maturity-buffer DQN baseline in stationary settings; (ii) positive effects under in-family parameter shifts, with significant gains in 5 of 10 shift scenarios after Holm--Bonferroni correction (up to 12.4\%); and (iii) consistent degradation under structural misspecification, where the choice model assumptions are violated (1.4--2.6\% lower revenue). These results characterize when partial behavioral models improve robustness under shift and when they introduce harmful bias.

Figures (4)

• Figure 1: Delayed feedback structure. Orders placed at earlier epochs generate shocks (modifications, cancellations) that resolve after 1--14 days. The DCM predicts shock outcomes from order characteristics, enabling immediate reward imputation.
• Figure 2: Replay timing. MB-DQN holds $(s_t,a_t,s_{t+1},r_t^{\mathrm{imm}})$ in a maturity buffer until $r_t^{\mathrm{del}}$ is revealed after $\Delta$ days. CA-DQN imputes $r_t^{\mathrm{del}}$ via DCM, enabling immediate replay insertion.
• Figure 3: Learning curves in stationary settings. Both MB-DQN (orange) and Choice-Assisted DQN (blue) converge to similar performance levels across all training durations (n=20 seeds per method, shaded regions show 95% confidence intervals). No significant differences are detected ($p > 0.05$ at all checkpoints), consistent with our theoretical prediction that choice-model assistance provides no advantage in stationary environments---the benefits emerge under distributional shift.
• Figure 4: Robustness under parameter shifts across 10 scenarios. Choice-Assisted DQN (blue bars) shows mixed results compared to MB-DQN (orange bars): significant improvements in 4 scenarios (up to +12.4% under low demand), significant underperformance in 2 scenarios (up to -9.6% under high competition), and no significant difference in 4 scenarios. These are parameter shifts that preserve the MNL structure---Section \ref{['subsec:out-of-family']} tests structural misspecification. Each bar represents mean revenue across 54 independent training runs with error bars showing standard error.

Theorems & Definitions (63)

• Definition 1: Choice-Model-Assisted RL
• Remark 1: Extension: Adaptive DCM
• Definition 2: Model-Imputed Sampling
• Theorem 3: Convergence under Fixed Choice Model
• Corollary 4: Asymptotic Neighborhood
• Remark 2: Extrapolation under Distributional Shift
• Remark 3: Clarification on A1 vs. A6
• Lemma 5: Conditional Unbiasedness of Doubly-Robust Target
• proof
• Lemma 6: Variance of Doubly-Robust Target