PLAS: Latent Action Space for Offline Reinforcement Learning

TL;DR

PLAS tackles offline RL by learning a policy in the latent action space of a CVAE, which naturally confines actions to the dataset's support and mitigates extrapolation errors. The method deterministically maps states to latent actions and decodes them into environment actions, with an optional bounded perturbation layer to enable controlled out-of-distribution generalization when beneficial. Empirical results on real robotic deformable-object manipulation and the D4RL benchmarks show competitive or superior performance to existing explicit-constraint offline RL methods, along with favorable Q-function estimations. The approach offers a simple, data-friendly alternative that leverages latent representations to balance in-distribution safety with potential out-of-distribution gains, making offline RL more practical for real-world robotics.

Abstract

The goal of offline reinforcement learning is to learn a policy from a fixed dataset, without further interactions with the environment. This setting will be an increasingly more important paradigm for real-world applications of reinforcement learning such as robotics, in which data collection is slow and potentially dangerous. Existing off-policy algorithms have limited performance on static datasets due to extrapolation errors from out-of-distribution actions. This leads to the challenge of constraining the policy to select actions within the support of the dataset during training. We propose to simply learn the Policy in the Latent Action Space (PLAS) such that this requirement is naturally satisfied. We evaluate our method on continuous control benchmarks in simulation and a deformable object manipulation task with a physical robot. We demonstrate that our method provides competitive performance consistently across various continuous control tasks and different types of datasets, outperforming existing offline reinforcement learning methods with explicit constraints. Videos and code are available at https://sites.google.com/view/latent-policy.

Figures (9)

• Figure 1: Overview: Instead of explicitly matching the action distribution of the agent policy with the behavior policy using divergence metrics such as KL or MMD, we implicitly constrain the policy to output actions within the support of the behavior policy through the latent action space.
• Figure 2: Network architecture for PLAS: Given a state, the latent policy outputs a latent action, which is then input into the decoder. The latent action space implicitly defines a constraint over the action output. An optional perturbation layer can be added on top of the output from the decoder to allow controlled generalization out of the training distribution.
• Figure 3: Real-robot experiment: (a) Experiment setup for the cloth sliding task. The cloth is fixed at the top left corner. (b) An example of the tactile sensor readings when the robot grasps at the edge of the cloth. (c) Training curves for the cloth sliding task on our method and the baselines. It shows the episode reward over five evaluation episodes every 150 training steps.
• Figure 4: Training performance for medium-expert and medium-replay datasets on locomotion tasks. Each curve is averaged over 3 seeds. Shaded area shows one standard deviation across seeds.
• Figure 5: We perform an analysis of Q-function errors of different methods, using the following metrics: (a) Mean-squared error of the Q-values (b) The percentage of overestimated Q-values (c) Mean of the positive errors (magnitude of overestimation) (d) Mean of the negative errors (magnitude of underestimation)