Geometry of Drifting MDPs with Path-Integral Stability Certificates

TL;DR

This work advances nonstationary reinforcement learning by modeling environment evolution as a differentiable homotopy path $M( au)$ and tracking the induced motion of the optimal Bellman fixed point $Q^*_ au$. It introduces a length--curvature--kink signature (PL, Curv, $oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{oldsymbol{}}}}}}}}}}}}$) to quantify drift, acceleration, and near-ties, and proves a solver-agnostic path-integral stability bound with gap-safe regions to certify local stability away from switches. Building on this, the paper derives first-/second-order tubes, ellipsoidal feasible regions, and projected tangent cones that characterize how $Q^*_ au$ can move along the path, enabling geometry-aware online control of learning/planning. It then delivers two lightweight, solver-agnostic wrappers, HT-RL and HT-MCTS, which estimate online proxies for PL, Curv, and kink proximity to adapt learning rates, target updates, regularization, and planning budgets, with formal dynamic-regret guarantees. Experiments on synthetic homotopy MDPs and deep-control benchmarks show improved tracking and reduced dynamic regret, particularly in oscillatory and switch-prone regimes, demonstrating practical impact for real-world nonstationary settings.

Abstract

Real-world reinforcement learning is often \emph{nonstationary}: rewards and dynamics drift, accelerate, oscillate, and trigger abrupt switches in the optimal action. Existing theory often represents nonstationarity with coarse-scale models that measure \emph{how much} the environment changes, not \emph{how} it changes locally -- even though acceleration and near-ties drive tracking error and policy chattering. We take a geometric view of nonstationary discounted Markov Decision Processes (MDPs) by modeling the environment as a differentiable homotopy path and tracking the induced motion of the optimal Bellman fixed point. This yields a length-curvature-kink signature of intrinsic complexity: cumulative drift, acceleration/oscillation, and action-gap-induced nonsmoothness. We prove a solver-agnostic path-integral stability bound and derive gap-safe feasible regions that certify local stability away from switch regimes. Building on these results, we introduce \textit{Homotopy-Tracking RL (HT-RL)} and \textit{HT-MCTS}, lightweight wrappers that estimate replay-based proxies of length, curvature, and near-tie proximity online and adapt learning or planning intensity accordingly. Experiments show improved tracking and dynamic regret over matched static baselines, with the largest gains in oscillatory and switch-prone regimes.

Figures (8)

• Figure 1: Synthetic ring MDP homotopy paths. Each panel (length-dominated, curvature-dominated, kink-prone) visualizes the moving reward bump and transition bias along $\tau$, together with the corresponding global action gap $g_\tau$. This figure illustrates how path length, curvature, and kink mass arise in simple MDPs.
• Figure 2: Synthetic ring MDP: validating the path--value bound and feasible tubes. Left: Theorem \ref{['thm:path-value']} tracks the true drift with a modest constant factor, and curvature/kink terms tighten the bound exactly when the corresponding geometric component is large. Right: The refined second-order tube is noticeably tighter on curvature-dominated paths, while both tubes closely envelope the true deviation on length-dominated paths.
• Figure 3: Deep control benchmarks under non-stationary homotopy paths (noisy drift). We show two representative environments (left: LunarLander, right: PointMass). Each panel reports average episode return vs. environment steps for static baselines and their HT-RL counterparts. Full learning curves for all environments are deferred to Appendix \ref{['app:exp-figures']}.
• Figure 4: Aggregate performance under non-stationary drift on a representative environment (LunarLander). Full 4-environment summaries are deferred to Appendix \ref{['app:exp-figures']}.
• Figure 5: Deep control benchmarks under clean non-stationary homotopy drift (no extra injected noise).Each panel reports average episode return versus environment steps for static baselines (dashed) and their HT-RL counterparts

Theorems & Definitions (47)

• Definition 3.1: Dual derivative and $\mathsf{W}_1^\ast$ norm
• Lemma 3.3: Sufficient conditions (Appendix \ref{['app:mixing']})
• Definition 4.1: Path length and curvature
• Definition 4.2: Global gap and kink set
• Definition 4.3: Kink penalty
• Lemma 5.1: Envelope property on the regular region
• Lemma 5.2: First-order homotopy derivative
• Lemma 5.3: Second-order homotopy derivative
• Theorem 5.4: Path integral value bound
• Theorem 5.5: First-order feasible tube