Exploring the Noise Resilience of Successor Features and Predecessor Features Algorithms in One and Two-Dimensional Environments

TL;DR

This work investigates the noise resilience of SF and PF learning in spatial navigation tasks modeled as Markov decision processes, comparing them against Q-learning and Q($\lambda$) baselines in both 1D and 2D noisy grid worlds. SF learning decomposes value into successor features and rewards, enabling transfer and robustness, while PF extends SF with eligibility traces to propagate credit across past states; the study analyzes how these approaches fare under Gaussian observation noise with levels $\sigma\in\{0.05,0.25,0.5\}$ and varying $\lambda$. Contrary to some prior expectations, PF does not consistently outperform SF in noisy environments; in 1D, SF shows superior robustness, whereas in 2D the noise effects are nonlinear and depend on $\lambda$. The findings bridge computational neuroscience and reinforcement learning by framing SF/PF in neurobiological terms and highlight practical implications for robotics and autonomous navigation, while underscoring the need for further exploration of parameter tuning and more complex environments.

Abstract

Based on the predictive map theory of spatial learning in animals, this study delves into the dynamics of Successor Feature (SF) and Predecessor Feature (PF) algorithms within noisy environments. Utilizing Q-learning and Q($λ$) learning as benchmarks for comparative analysis, our investigation yielded unexpected outcomes. Contrary to prevailing expectations and previous literature where PF demonstrated superior performance, our findings reveal that in noisy environments, PF did not surpass SF. In a one-dimensional grid world, SF exhibited superior adaptability, maintaining robust performance across varying noise levels. This trend of diminishing performance with increasing noise was consistent across all examined algorithms, indicating a linear degradation pattern. The scenario shifted in a two-dimensional grid world, where the impact of noise on algorithm performance demonstrated a non-linear relationship, influenced by the $λ$ parameter of the eligibility trace. This complexity suggests that the interaction between noise and algorithm efficacy is tied to the environmental dimensionality and specific algorithmic parameters. Furthermore, this research contributes to the bridging discourse between computational neuroscience and reinforcement learning (RL), exploring the neurobiological parallels of SF and PF learning in spatial navigation. Despite the unforeseen performance trends, the findings enrich our comprehension of the strengths and weaknesses inherent in RL algorithms. This knowledge is pivotal for advancing applications in robotics, gaming AI, and autonomous vehicle navigation, underscoring the imperative for continued exploration into how RL algorithms process and learn from noisy inputs.

Figures (11)

• Figure 1: Schematic representation of the environment-agent interaction in reinforcement learning with noisy observations. The agent receives an observation $\mathbf{o}_t$ and a reward $R_t$ at each time step $t$, based on which it decides on an action $a_t$. The environment, in turn, processes this action to update its state to $s_{t+1}$, providing the next state and reward ($r_{t+1}$) to the agent. A noise term $\boldsymbol{\epsilon}_t$ is added to the state representation $\phi(s_{t})$ before being fed back to the agent.
• Figure 2: Schematic of the grid world following the MDP. A. Illustrates a 1D grid world comprising 20 states, labeled from 1 to 20. The agent begins at state 1 (leftmost) and the goal is to reach state 20 (rightmost). B. A 2D grid world, structured as a 7x7 grid with barriers along the edges, restricting the agent's movement to a 5x5 area. The starting position is marked in the bottom right corner (triangle), and the goal is located in the top left corner (square).
• Figure 3: Cumulative Reward of RL Agents in a Noisy 1D Environment. A. Average cumulative rewards over 3000 episodes for Q-learning, Q($\lambda$)-learning, SF, and PF under different levels of observation noise ($\sigma$ = 0.05, 0.25, 0.5). Each line represents the mean cumulative reward across episodes, with the shaded area depicting the standard error of the mean. B. Distribution of cumulative rewards for each agent across noise settings. Box plots illustrate the median (central line), interquartile range (box limits), and outliers (individual points) for the cumulative rewards obtained over 3000 episodes, providing a comparative view of the reward distributions and the robustness of each algorithm to varying noise intensities.
• Figure 4: Episode Length Trends and Distributions for RL Agents in a 1D Noisy Environment. A. Demonstrates the decreasing trend in average episode length across 3000 episodes for various algorithms, under noise levels $\sigma$ of 0.05, 0.25, and 0.5. The shaded areas represent the standard error of the mean. B. Demonstrates the distribution of moving-averaged episode lengths in the final episodes, capturing the stabilization of learning across various noise levels. Box plots illustrate the median (central line), interquartile range (box limits), and outliers (individual points).
• Figure 5: Comparison of RL Algorithms for a Specific Range of Episode Lengths in a Noisy 1D Environment. This figure illustrates the frequency distribution of episode lengths across 3000 trials for various RL algorithms, categorized by noise levels with values of 0.05 (A), 0.25 (B), and 0.5 (C). The visualization concentrates on episodes ending with fewer than 22 steps or exceeding 99 steps. Box plot represent the distribution of the episode count for a given length, providing a visual comparison of algorithm efficiency and consistency.