DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learning

TL;DR

DROGO tackles the expensive computation of the Default Representation's (DR) principal eigenvector by learning it directly with a neural network. It retools the graph-drawing objective into a log-space formulation and uses natural-gradient updates, plus a terminal-state anchor, to reliably approximate $\log \mathbf e$ from transitions drawn under a default policy. Empirically, DROGO learns the DR eigenvector robustly across one-hot, coordinate, and pixel state representations and yields reward-shaped improvements over SR-based shaping in grid-world tasks. This approach scales DR-based methods to higher-dimensional observations and offers a practical path to reward-aware representations in reinforcement learning.

Abstract

In computational reinforcement learning, the default representation (DR) and its principal eigenvector have been shown to be effective for a wide variety of applications, including reward shaping, count-based exploration, option discovery, and transfer. However, in prior investigations, the eigenvectors of the DR were computed by first approximating the DR matrix, and then performing an eigendecomposition. This procedure is computationally expensive and does not scale to high-dimensional spaces. In this paper, we derive an objective for directly approximating the principal eigenvector of the DR with a neural network. We empirically demonstrate the effectiveness of the objective in a number of environments, and apply the learned eigenvectors for reward shaping.

Figures (9)

• Figure 1: The principal eigenvectors of the SR (top) and DR (bottom) in a ring-shaped environment with a low-region region at the top of the ring. The DR, unlike the SR, is reward-aware and captures the low-reward region in its principal eigenvector.
• Figure 2: The cosine similarity between the learned and ground-truth principal eigenvectors for different state representations averaged over 10 seeds. The shaded area indicates 95% bootstrap confidence interval.
• Figure 3: The principal eigenvector of the DR for (top) not treating terminal states as absorbing states tse2025rewardaware, and (bottom) treating terminal states as absorbing states in the four rooms environment. See Appendix \ref{['appendix:environments']} for a detailed explanation of the environments.
• Figure 4: $d_\mathscr{U}$ denotes the distance in the $\mathbf u$-space, which is assumed to be Euclidean. We aim to find the induced distance in the $\mathbf v$-space, denoted by $d_\mathscr{V}$, such that $d_\mathscr{V}$ between points in the $\mathbf v$-space is measured by $d_\mathscr{U}$ of their images under the $\exp$ transformation.
• Figure 5: The set of episodic environments used by tse2025rewardaware. From left to right: 1) grid task, 2) four rooms, 3) grid room, and 4) grid maze. The start state is in blue. The agent receives $-1$ reward at every time step unless it steps on red tiles ($-20$ reward) or reaches the goal in green ($0$ reward).

Theorems & Definitions (30)

• Proposition 3.1
• proof
• Proposition 3.2
• proof
• Proposition 3.3
• Proposition 3.4
• proof
• Proposition 3.5
• Corollary 3.6
• Proposition 3.7