Sequential memory improves sample and memory efficiency in Episodic Control

TL;DR

The paper tackles the data inefficiency of deep reinforcement learning by introducing Sequential Episodic Control (SEC), a hippocampus-inspired, non-parametric approach that stores and retrieves complete sequences of past state-action events. By imposing a sequential bias during memory retrieval and using a three-layer DAC architecture with STM/LTM, SEC achieves superior sample and memory efficiency compared to standard episodic controllers. Empirical results on Animal-AI foraging benchmarks show SEC outperforms baselines, with memory-constrained SEC providing substantial gains and forgetting mechanisms further stabilizing policy execution. These findings suggest that leveraging sequential, goal-oriented memories can dramatically enhance learning efficiency in embodied agents and may inform future hybrids with slower, model-based learners.

Abstract

State of the art deep reinforcement learning algorithms are sample inefficient due to the large number of episodes they require to achieve asymptotic performance. Episodic Reinforcement Learning (ERL) algorithms, inspired by the mammalian hippocampus, typically use extended memory systems to bootstrap learning from past events to overcome this sample-inefficiency problem. However, such memory augmentations are often used as mere buffers, from which isolated past experiences are drawn to learn from in an offline fashion (e.g., replay). Here, we demonstrate that including a bias in the acquired memory content derived from the order of episodic sampling improves both the sample and memory efficiency of an episodic control algorithm. We test our Sequential Episodic Control (SEC) model in a foraging task to show that storing and using integrated episodes as event sequences leads to faster learning with fewer memory requirements as opposed to a standard ERL benchmark, Model-Free Episodic Control, that buffers isolated events only. We also study the effect of memory constraints and forgetting on the sequential and non-sequential version of the SEC algorithm. Furthermore, we discuss how a hippocampal-like fast memory system could bootstrap slow cortical and subcortical learning subserving habit formation in the mammalian brain.

Figures (8)

• Figure 1: Sequential Episodic Control architecture. Following the Distributed Adaptive Control framework (Verschure2014 for review) SEC can be functionally divided into three layers: Reactive, Adaptive and Contextual. The reactive layer (green) implements a predefined random exploration algorithm. The adaptive layer (blue) acquires states of the world through a convolutional autoencoder, while the contextual layer (red) integrates a short- and long-term episodic memory buffer and an action-selection algorithm.
• Figure 2: Sequential Episodic Control memory storage and retrieval phases. During the storage phase, state-action couplets are stored in the short-term memory (STM) on a first-in, first-out basis at every timestep (top-left). Upon encountering a reward, the content of the STM is transferred to the long-term memory buffer (LTM), along with the reward value (top-right). During the retrieval phase, first, following Eq. \ref{['eq:eligibility']}, the current observed state is compared with the stored states in the LTM and the most similar ones are retrieved (middle). After that, following Eq. \ref{['eq:q_value']}, the action-value function for that given observed state is computed by taking the actions attached to the retrieved states along with their discounted relative reward value (bottom).
• Figure 3: Illustration of the Animal-AI benchmarks, showcasing both overhead and agent perspectives. The displayed environments include Double T-Maze (top-left), Cylinder (top-right), Object Permanence (bottom-left), and Detour (bottom-right). For each benchmark, the left side of the panel provides a third-person, bird's eye view of the environment, while the right side offers the first-person perspective as seen by the agent navigating the scenario.
• Figure 4: Comparative performance of Sequential Episodic Control (SEC) against several benchmark algorithms, namely Deep-Q Network (DQN), Model-Free Episodic Control (MFEC), Episodic Reinforcement Learning with Associative Memory (ERLAM), and non-sequential ablated version of SEC (NSEC). The presented results encompass four distinct Animal-AI benchmarks; the Double T-Maze (top-left), Cylinder (top-right), Object Permanence (bottom-left), and Detour (bottom-right) tasks. For clarity and statistical robustness, average performance metrics were calculated using a sliding window encompassing 20 episodes (20.000 frames). The error bars denote the standard error (SE) to provide a measure of the variability in the dataset.
• Figure 5: Effect of memory constraints across episodes between SEC and NSEC models in terms of reward accumulation and entropy in the Double T-Maze benchmark. Top panels: Mean reward per episode accumulated by SEC (left) and Non-SEC models (right). Bottom panels: Mean entropy on the episodic policy, computed as the average of the entropies of the probability distributions derived from $\ \widehat{Q}_{s,a} ^{LTM}$ at every timestep of the episode. Vertical bars represent the average episode around which the memory was filled. Average values were computed using a sliding window of 20 episodes. Error bars represent SE.