Learning Optimal Distributionally Robust Stochastic Control in Continuous State Spaces

TL;DR

This work introduces distributionally robust stochastic control (DRSC) for infinite-horizon, possibly continuous-state problems, integrating adaptive adversarial perturbations to environment inputs to improve policy reliability. It develops two adversary models, current-action-aware (CAA) and current-action-unaware (CAU), and proves robust Bellman equations have unique solutions, preserving stationary optimality. The authors establish minimax learning rates for uniformly estimating the robust value under Wasserstein and $f_k$-divergence ambiguity sets, show these rates are dimensionally efficient, and provide matching lower bounds. They design practical actor–critic algorithms that leverage strong duality to handle the robust inner problems, enabling scalable learning in both CAA and CAU settings, including continuous-action extensions. Demonstrations on inventory control with real data and portfolio optimization illustrate robust policies outperform non-robust baselines and highlight the nuanced performance trade-offs between CAA and CAU under varying distributional shifts.

Abstract

We study data-driven learning of robust stochastic control for infinite-horizon systems with potentially continuous state and action spaces. In many managerial settings--supply chains, finance, manufacturing, services, and dynamic games--the state-transition mechanism is determined by system design, while available data capture the distributional properties of the stochastic inputs from the environment. For modeling and computational tractability, a decision maker often adopts a Markov control model with i.i.d. environment inputs, which can render learned policies fragile to internal dependence or external perturbations. We introduce a distributionally robust stochastic control paradigm that promotes policy reliability by introducing adaptive adversarial perturbations to the environment input, while preserving the modeling, statistical, and computational tractability of the Markovian formulation. From a modeling perspective, we examine two adversary models--current-action-aware and current-action-unaware--leading to distinct dynamic behaviors and robust optimal policies. From a statistical learning perspective, we characterize optimal finite-sample minimax rates for uniform learning of the robust value function across a continuum of states under ambiguity sets defined by the $f_k$-divergence and Wasserstein distance. To efficiently compute the optimal robust policies, we further propose algorithms inspired by deep reinforcement learning methodologies. Finally, we demonstrate the applicability of the framework to real managerial problems.

Figures (4)

• Figure 1: Robust value functions with various sizes of the uncertainty set.
• Figure 2: Robust policies with various sizes of the uncertainty set.
• Figure 3: Value difference between non-robust optimal policy and two robust policies.
• Figure 4: Robust policies for one store in the inventory control problem.

Theorems & Definitions (31)

• Definition 1: CAA Robust Bellman Equation
• Definition 2: CAU Robust Bellman Equation
• Remark
• Theorem 1
• Remark
• Example 1: Service or manufacturing systems
• Example 2: Inventory control
• Example 3: Portfolio management
• Definition 3: Wasserstein distance ambiguity sets
• Definition 4: $f_k$-divergence ambiguity sets