Toward Explainable Offline RL: Analyzing Representations in Intrinsically Motivated Decision Transformers

TL;DR

This work addresses interpretability in offline RL by analyzing how intrinsic motivation reshapes embeddings in Elastic Decision Transformers. It introduces a post-hoc explainability framework that uses embedding-geometry metrics and Pearson correlations to relate representations to performance, and studies two RND-based intrinsic-motivation variants, yielding total objective $L_{ ext{total}} = L_{ ext{action}} + eta L_{ ext{state}} + eta' L_{ ext{exp}} + eta'' L_{ ext{ret}} + L_{ ext{int}}$ with $L_{ ext{int}}$ defined as $L_{ ext{int}} = ig Vert f_{ ext{pred}}(x; heta_{ ext{pred}}) - f_{ ext{target}}(x; heta_{ ext{target}}) ig Vert_2^2$; and showing that intrinsic motivation yields environment-specific embedding geometries. Empirically, EDT-SIL and EDT-TIL provide complementary representational advantages, with a 3-layer RND predictor often optimal. These findings suggest intrinsic motivation acts as a representational prior, shaping embedding geometry to support more effective policy learning in offline settings and offering guidance for interpretable design of offline RL systems.

Abstract

Elastic Decision Transformers (EDTs) have proved to be particularly successful in offline reinforcement learning, offering a flexible framework that unifies sequence modeling with decision-making under uncertainty. Recent research has shown that incorporating intrinsic motivation mechanisms into EDTs improves performance across exploration tasks, yet the representational mechanisms underlying these improvements remain unexplored. In this paper, we introduce a systematic post-hoc explainability framework to analyze how intrinsic motivation shapes learned embeddings in EDTs. Through statistical analysis of embedding properties (including covariance structure, vector magnitudes, and orthogonality), we reveal that different intrinsic motivation variants create fundamentally different representational structures. Our analysis demonstrates environment-specific correlation patterns between embedding metrics and performance that explain why intrinsic motivation improves policy learning. These findings show that intrinsic motivation operates beyond simple exploration bonuses, acting as a representational prior that shapes embedding geometry in biologically plausible ways, creating environment-specific organizational structures that facilitate better decision-making.

Figures (2)

• Figure 1: Architecture of the Elastic Decision Transformer with intrinsic motivation mechanisms proposed in esann2025.
• Figure 2: Cumulative Human-Normalized Scores (HNS) obtained by the best models trained on each environment of the Medium dataset. 3-layer variants (red borders) achieve optimal performance for both SIL and TIL mechanisms.