Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents

TL;DR

Fast adaptation to unseen tasks is crucial for embodied agents, and off-policy meta-RL relies on efficient data sampling to build task representations. The authors compare two representative off-policy meta-RL models—PEARL based on Thompson sampling and SAC, and VariBAD based on Bayes-optimality with a VAE—under long- and short-memory data sampling regimes across MuJoCo control tasks and sparse-reward navigation tasks. They find that Bayes-optimal VariBAD yields more robust adaptation to unknown dynamics and sparse rewards, while PEARL's performance and exploration are highly sensitive to memory-length sampling. These results highlight the critical role of data sampling strategies in shaping task representations and online adaptability, suggesting that Bayesian-context approaches offer stronger robustness in real-world, sparse-reward settings.

Abstract

Fast adaptation to new tasks is extremely important for embodied agents in the real world. Meta-reinforcement learning (meta-RL) has emerged as an effective method to enable fast adaptation in unknown environments. Compared to on-policy meta-RL algorithms, off-policy algorithms rely heavily on efficient data sampling strategies to extract and represent the historical trajectories. However, little is known about how different data sampling methods impact the ability of meta-RL agents to represent unknown environments. Here, we investigate the impact of data sampling strategies on the exploration and adaptability of meta-RL agents. Specifically, we conducted experiments with two types of off-policy meta-RL algorithms based on Thompson sampling and Bayes-optimality theories in continuous control tasks within the MuJoCo environment and sparse reward navigation tasks. Our analysis revealed the long-memory and short-memory sequence sampling strategies affect the representation and adaptive capabilities of meta-RL agents. We found that the algorithm based on Bayes-optimality theory exhibited more robust and better adaptability than the algorithm based on Thompson sampling, highlighting the importance of appropriate data sampling strategies for the agent's representation of an unknown environment, especially in the case of sparse rewards.

Figures (10)

• Figure 1: Motivations of our work.
• Figure 2: PEARL and off-policy VariBAD famework.
• Figure 3: Long memory sequence sampling and short memory sequence sampling. Different context sampling strategies can lead to shifts in the distribution of task representations, thereby affecting the agent's exploration and adaptation capabilities.
• Figure 4: Tasks training. Dashed lines correspond to the maximum return achieved by PEARL after 1e6 steps. Solid lines correspond to average return achieved by VariBAD. In the Ant-Semicircle and Half-Cheetah-Vel tasks, PEARL and VariBAD converge to similar average returns. However, in the Sparse-Point-Robot task, VariBAD significantly outperforms PEARL.
• Figure 5: The average return during the meta-training phase of PEARL and off-policy VariBAD after using short and long memory sampling strategies.