Statistical Reinforcement Learning in the Real World: A Survey of Challenges and Future Directions

TL;DR

This survey reframes real-world reinforcement learning as a three-component process—within-deployment online learning, between-deployment offline analysis, and sequential redeployments for continual improvement—to address practical constraints of data scarcity and nonstationary environments. It synthesizes advances in statistical RL across online learning, offline inference, and deployment design, with emphasis on adaptive experiments, adaptive interventions, and causal reasoning to improve data efficiency and generalization. The paper highlights the challenges of misspecification, pooling-induced dependencies, and nonstationarity while underscoring the need for practical guidelines, benchmarks, and LLM-enabled approaches to bridge theory and real-world impact. By advocating use-inspired research that prioritizes replicability, valid inference, and continual improvement, the work outlines a roadmap for making RL more reliable and impactful in domains like digital health and education.

Abstract

Reinforcement learning (RL) has achieved remarkable success in real-world decision-making across diverse domains, including gaming, robotics, online advertising, public health, and natural language processing. Despite these advances, a substantial gap remains between RL research and its deployment in many practical settings. Two recurring challenges often underlie this gap. First, many settings offer limited opportunity for the agent to interact extensively with the target environment due to practical constraints. Second, many target environments often undergo substantial changes, requiring redesign and redeployment of RL systems (e.g., advancements in science and technology that change the landscape of healthcare delivery). Addressing these challenges and bridging the gap between basic research and application requires theory and methodology that directly inform the design, implementation, and continual improvement of RL systems in real-world settings. In this paper, we frame the application of RL in practice as a three-component process: (i) online learning and optimization during deployment, (ii) post- or between-deployment offline analyses, and (iii) repeated cycles of deployment and redeployment to continually improve the RL system. We provide a narrative review of recent advances in statistical RL that address these components, including methods for maximizing data utility for between-deployment inference, enhancing sample efficiency for online learning within-deployment, and designing sequences of deployments for continual improvement. We also outline future research directions in statistical RL that are use-inspired -- aiming for impactful application of RL in practice.

Figures (4)

• Figure 1: Reinforcement learning (RL) in practice, framed as a three-component process. The process involves online learning and optimization within each deployment, offline learning and inference between deployments, and a continual deployment-redeployment feedback process for continual improvement.
• Figure 2: Illustrative causal diagrams of sequential decision-making environments: (a) contextual bandit, (b) Markov decision process (MDP), and (c) partially observable MDP (POMDP). Note that the action at time $t$ is selected by the agent and may depend on the entire history of past observations. In the MDP diagram (b), the red arrow highlights the effect of the action $A_t$ on the next state $S_{t+1}$, which is absent in the contextual bandit setting (a). In the POMDP diagram (c), the blue arrows indicate the additional observation process, whereby the agent observes $O_t$ rather than the latent state $S_t$; dotted squares indicate unobserved (latent) states. Other examples of causal diagrams for the environment include those used in causal RL; see, e.g., bareinboim2021introductionge2025review.
• Figure 3: An example of a causal DAG that describes the HeartSteps environment. HeartSteps is a JITAI described in Section \ref{['sec_realWorld']} that aims to help users increase and maintain physical activity. The action $A_{d, k}$ refers to whether a walking suggestion is sent or not. Each day $d$ contains $K$ interventions. The reward $R_d$ is the daily commitment to being active. The engagement $E_d$ represents the user's daily engagement with the app. The context can include the current activity status of the user, the location of the user, etc. The variables $M_{d, k}$ and $N_{d, k}$ are the mediators between $A_{d, k}$ and $R_d, E_d$.
• Figure 4: Pooling algorithm. Depiction of how a pooling algorithm combines or "pools" data across individuals to form and update the policies over time. Adapted from zhang2024replicable.