Structuring Value Representations via Geometric Coherence in Markov Decision Processes

TL;DR

We frame value learning in reinforcement learning as constructing a target poset $(\bar X,\preceq)$ over state–action pairs, where the symmetry quotient $\bar X=X/\sim$ and poset refinements $\preceq_k$ enforce reflexivity, transitivity, and antisymmetry to approach the optimal $Q^*$. The proposed Geometric Coherence Regularized RL (GCR-RL) yields two realizations: a soft coherence path that regularizes TD learning using near-equivariance losses and a differentiable isotonic projection on a TD-derived DAG, and a hard, lossless manifold enforcement that updates along a tangent space while contracting infeasible directions; both routes come with convergence-type guarantees and variance-reduction effects. The framework jointly learns near-equivariant transforms with action relabeling and constructs TD-informed partial orders to stabilize value propagation, achieving significant improvements in sample efficiency and stability across symmetry-rich grids, action-permutation tasks, Atari, and non-transitive chain tasks. These results demonstrate that encoding geometric coherence in value representations supports faster, more robust RL in complex environments, with potential for scalable online DAG construction and extensions to continuous control and planning.

Abstract

Geometric properties can be leveraged to stabilize and speed reinforcement learning. Existing examples include encoding symmetry structure, geometry-aware data augmentation, and enforcing structural restrictions. In this paper, we take a novel view of RL through the lens of order theory and recast value function estimates into learning a desired poset (partially ordered set). We propose \emph{GCR-RL} (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements -- by refining posets in previous steps and learning additional order relationships from temporal difference signals -- thus ensuring geometric coherence across the sequence of posets underpinning the learned value functions. Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements. Their theoretical properties and convergence rates are analyzed. We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.

Figures (5)

• Figure 1: Representative learning curves for GCR-RL and baselines. Full per-environment curves (including Atari and Minigrid) appear in Appendix \ref{['app:curves']}.
• Figure 2: Learning curves for GCR-RL and baselines across all environments.
• Figure 3: AUC@N (M1) for all environments. Each subfigure shows a per-environment bar plot over algorithms.
• Figure 4: Steps-to-threshold (M1) across environments. Each subfigure shows the number of steps needed to reach a fixed score.
• Figure 5: Final-return distributions (M2/M3) across environments. Each subfigure is a boxplot over seeds for each algorithm.

Theorems & Definitions (79)

• Definition 2.1: $\mathcal{G}$-invariance
• Theorem 3.1: Equivariance under invariance
• Theorem 3.2: Effective sample amplification
• Theorem 3.3: Greedy DAGification
• Theorem 3.4: Penalty-method consistency
• Theorem 3.5: Order-consistency on a DAG
• Theorem 3.6: Residual decrease under order alignment
• Theorem 3.7: Stability improvement
• Theorem 3.8: Bias--variance trade-off
• Theorem 3.9: Identifiability of near-equivariance