Continual Reinforcement Learning via Autoencoder-Driven Task and New Environment Recognition

TL;DR

The paper tackles continual reinforcement learning without external task signals or data replay by coupling a growing set of policy networks with per-task autoencoders that detect novelty and match observations to known environments. Observations are processed by all autoencoders; a novel environment triggers the creation of a new policy-autoencoder pair, while matches select the best-fitting past policy for retrieval. Across Minigrid and Atari domains, the approach preserves prior knowledge as new tasks are learned and outperforms a vanilla single-policy baseline that suffers catastrophic forgetting. The work demonstrates that autoencoder-driven recognition enables autonomous, scalable continual RL with explicit environment discrimination, offering a pathway to integrate with other continual-learning strategies while reducing memory and signaling requirements. This has practical implications for deploying agents in dynamic, real-world settings where task boundaries are not predefined or signaled.

Abstract

Continual learning for reinforcement learning agents remains a significant challenge, particularly in preserving and leveraging existing information without an external signal to indicate changes in tasks or environments. In this study, we explore the effectiveness of autoencoders in detecting new tasks and matching observed environments to previously encountered ones. Our approach integrates policy optimization with familiarity autoencoders within an end-to-end continual learning system. This system can recognize and learn new tasks or environments while preserving knowledge from earlier experiences and can selectively retrieve relevant knowledge when re-encountering a known environment. Initial results demonstrate successful continual learning without external signals to indicate task changes or reencounters, showing promise for this methodology.

Figures (8)

• Figure 1: Our two experimental domains: Minigrid environment subtypes MinigridMiniworld23 (a-c) and Atari games towers2024gymnasium (d-f) to be learned incrementally.
• Figure 2: Overview of our system design. The system at any time is composed of a number of policy networks (three in this example) and an autoencoder (AE) associated with each of them. (1) The agent obtains observations from the environment (as images in our implementation). (2) The observations are passed as input to all autoencoders of all policy networks available for them to attempt reconstruction. The error on each autoencoder's reconstruction is computed. (3) If there are autoencoders whose reconstruction errors are below automatically estimated reconstruction thresholds (see main text), then the policy network associated with the autoencoder with the lowest reconstruction error is chosen (in this figure, that's Policy 3, meaning that AE-3 provided the lowest reconstruction error). If all reconstructions have errors above the threshold, this is interpreted as the observation of an unrecognized environment. A new policy network and a new associated autoencoder are created for training on this new task. (The illustrative reconstructions on this figure are actual in-operation outputs by our system, for tasks corresponding to Minigrid's Dynamic-Obstacles, Key-Door and Lava-Gap environments.)
• Figure 3: Learning Flow 1 (Retrospective performance)
• Figure 4: Learning Flow 2 (Ongoing performance)
• Figure 5: Normalized rewards obtained by Vanilla agent during retrospective performance evaluation at the three tasks. Labels T1, T2 and T3 on x-axis signify the time of training on the corresponding task, followed by subsequent evaluation on tasks up to that point. The plot shows clear destructive adaptation as performance in prior tasks are not retained.