Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics

TL;DR

This work reframes TD-based RL under non-Markov dynamics through a topological lens, treating TD errors as a $1$-cochain and separating integrable (Markovable) from non-integrable (topological residual) components via a Bellman–de Rham projection. It introduces a Poisson-based characterization of the optimal potential and a practical HodgeFlow Policy Search (HFPS) algorithm that updates using only the integrable TD signal while using the residual as a diagnostic of Bellman mismatch. Theoretical results establish consistency, stability, and sensitivity of the decomposition, and experiments on synthetic and control tasks show HFPS improves stability and robustness in non-Markovian settings. Overall, the paper provides a principled, geometry-inspired framework for TD learning that highlights when standard Bellman updates suffice and how to adapt when topological non-integrability arises, with practical algorithms and diagnostics for real-world non-Markovian RL. The approach has potential implications for offline RL, partial observability, and environments with memory effects, guiding both theory and algorithm design.

Abstract

Non-Markovian dynamics are commonly found in real-world environments due to long-range dependencies, partial observability, and memory effects. The Bellman equation that is the central pillar of Reinforcement learning (RL) becomes only approximately valid under Non-Markovian. Existing work often focus on practical algorithm designs and offer limited theoretical treatment to address key questions, such as what dynamics are indeed capturable by the Bellman framework and how to inspire new algorithm classes with optimal approximations. In this paper, we present a novel topological viewpoint on temporal-difference (TD) based RL. We show that TD errors can be viewed as 1-cochain in the topological space of state transitions, while Markov dynamics are then interpreted as topological integrability. This novel view enables us to obtain a Hodge-type decomposition of TD errors into an integrable component and a topological residual, through a Bellman-de Rham projection. We further propose HodgeFlow Policy Search (HFPS) by fitting a potential network to minimize the non-integrable projection residual in RL, achieving stability/sensitivity guarantees. In numerical evaluations, HFPS is shown to significantly improve RL performance under non-Markovian.

Figures (8)

• Figure 1: Synthetic validation of the Bellman decomposition. Left: tabular ring MDP (integrable vs. non-integrable). Right: random-feature MDP (MSVE and residual behavior).
• Figure 2: Episode returns (Nonmarkov). Return versus environment steps (mean $\pm$ one standard deviation over seeds) for HFPS and baselines on two representative control tasks.
• Figure 3: Episode returns. Return versus environment steps (mean $\pm$ one standard deviation over seeds) for HFPS and baselines on five control tasks (columns) under Nonmarkov
• Figure 4: Episode returns. Return versus environment steps (mean $\pm$ one standard deviation over seeds) for HFPS and baselines on five control tasks (columns) under three observation regimes (rows): Clean, Noisy, and Sticky.
• Figure 5: Cumulative AUC trajectories Each panel plots the cumulative integral of the corresponding return curve in Figure \ref{['fig:control-returns']}. Higher curves indicate better overall sample efficiency under the same training budget.

Theorems & Definitions (33)

• Definition 2.1: Discounted triplet occupancy measure
• Definition 2.2: Cochain spaces
• Definition 2.3: Discrete de Rham differential
• Lemma 2.4: Boundedness and adjoint of $d$
• Definition 2.5: Zero-th order Hodge Laplacian
• Definition 3.1: Value function and TD error
• Definition 3.2: Exact 1-cochains
• Definition 3.3: Topologically integrable value function
• Theorem 3.4: Hodge-type decomposition in $C^1$
• Corollary 3.5: Topological decomposition of TD error