Action-Sufficient Goal Representations

TL;DR

This work analyzes how goal representations affect hierarchical, offline goal-conditioned RL. It introduces action sufficiency, an information-theoretic criterion I(A; G | S, Z) = 0, and proves that value sufficiency does not imply action sufficiency, showing that value-based encodings can discard action-critical distinctions. Through exact analysis in the Discrete Cube and experiments on HIQL with OGBench cube tasks, it demonstrates that actor-based goal representations learned via end-to-end log-loss induce approximate action sufficiency and yield superior control performance over value-based representations. The findings advocate for integrating the actor's objective into representation learning to preserve the action-relevant information necessary for robust, long-horizon control in offline GCRL, with implications for designing interfaces between planning and control components.

Abstract

Hierarchical policies in offline goal-conditioned reinforcement learning (GCRL) addresses long-horizon tasks by decomposing control into high-level subgoal planning and low-level action execution. A critical design choice in such architectures is the goal representation-the compressed encoding of goals that serves as the interface between these levels. Existing approaches commonly derive goal representations while learning value functions, implicitly assuming that preserving information sufficient for value estimation is adequate for optimal control. We show that this assumption can fail, even when the value estimation is exact, as such representations may collapse goal states that need to be differentiated for action learning. To address this, we introduce an information-theoretic framework that defines action sufficiency, a condition on goal representations necessary for optimal action selection. We prove that value sufficiency does not imply action sufficiency and empirically verify that the latter is more strongly associated with control success in a discrete environment. We further demonstrate that standard log-loss training of low-level policies naturally induces action-sufficient representations. Our experimental results a popular benchmark demonstrate that our actor-derived representations consistently outperform representations learned via value estimation.

Figures (3)

• Figure 1: Left: The results of low-level policy evaluation on short-horizon subgoal $s_{t+k}$ from optimal trajectory $\tau^{\star}$. Right: The visualization of value function $V(s,g)$ by varying $s$ from initial state to goal $g$ in $\tau^{\star}$. Note that though the value function effectively captures the discounted temporal distance throughout all environments, the low-level policy equipped with $\phi_V$ achieves much lower success rate than the counterpart which uses $\phi_A$.
• Figure 2: The Discrete Cube envorinment. This domain mirrors the structure of pick-and-place tasks while allowing for the exact computation of information-theoretic quantities. We use this tractability to rigorously verify that control success depends on preserving action sufficiency rather than value sufficiency.
• Figure 3: Discrete Cube Result.(a) Control success rate plotted over the $(\Delta_V,\Delta_A)$ plane for representations $\phi$, evaluated using the mixed policy $\pi_\phi(a \mid s, z)$. (b) For (near-) value-sufficient representations ($\Delta_V<0.2$), control success rate plotted against $I(A;Z\mid S,V)$, with points colored by $\Delta_A$.

Theorems & Definitions (29)

• Proposition 4.1: Conditional KL Risk Decomposition
• Definition 4.2: Action-Sufficient Representation
• Definition 5.1: Value-Sufficient Representation
• Proposition 5.2: Action Information Decomposition
• Lemma 7.1: Decomposition of Actor NLL
• Theorem 7.2: Near-Optimal Actor NLL Implies Approximate Action Sufficiency
• Definition 3.2: Strict value sufficiency
• Lemma 3.3: Injectivity implies information equivalence
• proof
• Proposition 3.4: Strict value sufficiency implies $\sigma(G)=\sigma(Z)$