Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli

Abstract

Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space. A recent framework, which we refer to as Action-based Policy Compression (APC), mitigates this issue by compressing the parameter space $Θ$ into a low-dimensional latent manifold $\mathcal Z$ using a learned generative mapping $g:\mathcal Z \to Θ$. However, its performance is severely constrained by relying on immediate action-matching as a reconstruction loss, a myopic proxy for behavioral similarity that suffers from compounding errors across sequential decisions. To overcome this bottleneck, we introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage. Specifically, we propose two principal improvements: (1) we curate the dataset generation with an information-theoretic uniqueness metric that delivers a diverse population of policies; and (2) we propose a fully differentiable compression objective that directly minimizes the divergence between the true and reconstructed mixture occupancy distributions. These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors while retaining most of the original parameter space's expressivity. Finally, we empirically validate the advantages of our contributions across multiple continuous control benchmarks.

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

Abstract

Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space. A recent framework, which we refer to as Action-based Policy Compression (APC), mitigates this issue by compressing the parameter space into a low-dimensional latent manifold using a learned generative mapping . However, its performance is severely constrained by relying on immediate action-matching as a reconstruction loss, a myopic proxy for behavioral similarity that suffers from compounding errors across sequential decisions. To overcome this bottleneck, we introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage. Specifically, we propose two principal improvements: (1) we curate the dataset generation with an information-theoretic uniqueness metric that delivers a diverse population of policies; and (2) we propose a fully differentiable compression objective that directly minimizes the divergence between the true and reconstructed mixture occupancy distributions. These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors while retaining most of the original parameter space's expressivity. Finally, we empirically validate the advantages of our contributions across multiple continuous control benchmarks.

Paper Structure

This paper contains 24 sections, 18 equations, 12 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Background: Action-based Policy Compression
  4. Methodology: Occupancy-based Policy Compression
  5. Experiments
  6. Policy Dataset Curation
  7. Latent Behavior Compression
  8. Latent Behavior Optimization
  9. Conclusion
  10. Extended Related Work
  11. Policy Dataset Curation: Derivations and Estimators
  12. Derivation of the Uniqueness Score
  13. Tractable Estimation via GMMs
  14. KL Divergence Bound and the Saturation Effect
  15. Numerically Stable Importance Weight Computation
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Comparison of APC and OPC thresholding in the RC environment: reward distribution in the curated datasets (top row, frequency is on a log scale) and score distribution comparison (bottom row, scores are min-max normalized). The vertical dashed line marks the 5% cut-off.
  • Figure 2: Ablation study comparing Latent Space topologies in the HP environment. Rows represent reward functions, while columns represent different score/loss combinations. Lighter and darker colors indicate higher and lower returns of the decoded policy.
  • Figure 3: Performance comparison in MC, RC, and HP across different tasks. We report the average and 95% confidence interval over 10 runs of the best-performing algorithms for each task.
  • Figure 4: Reward distributions comparison in MC, RC, and HP.
  • Figure 5: Correlation between APC (top row) and OPC (bottom row) scores and rewards in MC, RC, and HP. The vertical dashed line indicates the top 5% selection threshold.
  • ...and 7 more figures