Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing Discrete Representations For Continual Reinforcement Learning

Edan Meyer, Adam White, Marlos C. Machado

TL;DR

This paper investigates the impact of discrete latent representations on reinforcement learning, focusing on world-model learning and continual RL. By comparing vanilla autoencoders, sparse FTA encodings, and discrete VQ-VAE representations within decoupled encoder architectures, the study shows that discrete, particularly multi-one-hot, representations enable more accurate world models with less capacity and accelerate adaptation in non-stationary environments. The analysis reveals that the gains largely stem from the representation design (sparse, binary encodings) rather than mere discreteness, with strong evidence that discrete representations facilitate rapid policy learning and faster adaptation in continual RL. These findings suggest that carefully crafted discrete representations can yield practical benefits for scalable, continual RL in large or evolving environments.

Abstract

Reinforcement learning (RL) agents make decisions using nothing but observations from the environment, and consequently, heavily rely on the representations of those observations. Though some recent breakthroughs have used vector-based categorical representations of observations, often referred to as discrete representations, there is little work explicitly assessing the significance of such a choice. In this work, we provide a thorough empirical investigation of the advantages of representing observations as vectors of categorical values within the context of reinforcement learning. We perform evaluations on world-model learning, model-free RL, and ultimately continual RL problems, where the benefits best align with the needs of the problem setting. We find that, when compared to traditional continuous representations, world models learned over discrete representations accurately model more of the world with less capacity, and that agents trained with discrete representations learn better policies with less data. In the context of continual RL, these benefits translate into faster adapting agents. Additionally, our analysis suggests that the observed performance improvements can be attributed to the information contained within the latent vectors and potentially the encoding of the discrete representation itself.

Harnessing Discrete Representations For Continual Reinforcement Learning

TL;DR

This paper investigates the impact of discrete latent representations on reinforcement learning, focusing on world-model learning and continual RL. By comparing vanilla autoencoders, sparse FTA encodings, and discrete VQ-VAE representations within decoupled encoder architectures, the study shows that discrete, particularly multi-one-hot, representations enable more accurate world models with less capacity and accelerate adaptation in non-stationary environments. The analysis reveals that the gains largely stem from the representation design (sparse, binary encodings) rather than mere discreteness, with strong evidence that discrete representations facilitate rapid policy learning and faster adaptation in continual RL. These findings suggest that carefully crafted discrete representations can yield practical benefits for scalable, continual RL in large or evolving environments.

Abstract

Reinforcement learning (RL) agents make decisions using nothing but observations from the environment, and consequently, heavily rely on the representations of those observations. Though some recent breakthroughs have used vector-based categorical representations of observations, often referred to as discrete representations, there is little work explicitly assessing the significance of such a choice. In this work, we provide a thorough empirical investigation of the advantages of representing observations as vectors of categorical values within the context of reinforcement learning. We perform evaluations on world-model learning, model-free RL, and ultimately continual RL problems, where the benefits best align with the needs of the problem setting. We find that, when compared to traditional continuous representations, world models learned over discrete representations accurately model more of the world with less capacity, and that agents trained with discrete representations learn better policies with less data. In the context of continual RL, these benefits translate into faster adapting agents. Additionally, our analysis suggests that the observed performance improvements can be attributed to the information contained within the latent vectors and potentially the encoding of the discrete representation itself.
Paper Structure (22 sections, 5 equations, 16 figures, 4 tables, 2 algorithms)

This paper contains 22 sections, 5 equations, 16 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Algorithms
  4. Environments
  5. World-Model Learning with Discrete Representations
  6. Learning World Models
  7. Experiments
  8. Model Rollouts
  9. Scaling the World Model
  10. Representation Matters
  11. Model-Free RL with Discrete Representations
  12. Episodic RL
  13. Continual RL
  14. Conclusion & Future Work
  15. Autoencoders Explained
  16. ...and 7 more sections

Figures (16)

  • Figure 1: Minigrid environments used in our experiments. We refer to these as the (a) empty, (b) crossing, and (c) door key environments. The agent receives lower-resolution RGB arrays representing pixels as observations.
  • Figure 2: The mean KL divergence between the ground-truth and the world model induced state distributions. Lower values are better, indicating a closer imitation of the real environment dynamics. The VQ-VAE and Vanilla AE learn near-perfect models in the empty environment, so the curves are so close to zero that they are not visible without maginification. FTA AE and End-to-End experiments were not run in the empty environment because of the triviality. Curves depict a 95% confidence intervals over 20 runs.
  • Figure 3: The median KL divergence between the ground-truth and the world model induced state distributions, averaged over 30 steps. Lower is better, indicating a closer imitation of the real environment dynamics. The x-axis gives the number of hidden units per layer for all three layers of the world model. The shaded region depicts a 95% confidence interval over 20 runs. Error bars are wide for the end-to-end method due to a few divergent runs. Training the end-to-end model is harder because gradients for multiple objectives must be passed back in time through multiple steps.
  • Figure 4: The mean KL divergence between the ground-truth and the world model induced state distributions. Lower values are better, indicating a closer imitation of the real environment dynamics. Both methods use the same VQ-VAE architecture, but represent the information in different ways. Curves depict 95% confidence intervals over 20 runs.
  • Figure 5: Performance of RL agents as measured by episode length with a 95% confidence interval over 30 runs. Lower is better. (a-b) Agents are trained with PPO and autoencoder objectives from the beginning. (c-d) The PPO objective is introduced only after the dotted line (with the exception of the end-to-end method).
  • ...and 11 more figures