Tuning the Weights: The Impact of Initial Matrix Configurations on Successor Features Learning Efficacy

TL;DR

This work investigates how initial SF weight configurations influence learning efficiency in grid-world reinforcement learning. By comparing identity, zero, and small random initializations (including Xavier and He methods), the study demonstrates that random initialization accelerates convergence to the optimal SR place-field, speeds value-learning, and stabilizes step-length reduction, especially in larger grids. PCA and L1-distance analyses reveal distinct learning trajectories for random initializations, suggesting that initial synaptic weights can bias the evolution of the SF representation. The findings offer insights for designing more efficient SF-based agents and deepen the connection between SF learning and hippocampal place-cell representations in neuroscience.

Abstract

The focus of this study is to investigate the impact of different initialization strategies for the weight matrix of Successor Features (SF) on learning efficiency and convergence in Reinforcement Learning (RL) agents. Using a grid-world paradigm, we compare the performance of RL agents, whose SF weight matrix is initialized with either an identity matrix, zero matrix, or a randomly generated matrix (using Xavier, He, or uniform distribution method). Our analysis revolves around evaluating metrics such as value error, step length, PCA of Successor Representation (SR) place field, and the distance of SR matrices between different agents. The results demonstrate that RL agents initialized with random matrices reach the optimal SR place field faster and showcase a quicker reduction in value error, pointing to more efficient learning. Furthermore, these random agents also exhibit a faster decrease in step length across larger grid-world environments. The study provides insights into the neurobiological interpretations of these results, their implications for understanding intelligence, and potential future research directions. These findings could have profound implications for the field of artificial intelligence, particularly in the design of learning algorithms.

Figures (8)

• Figure 1: Schematic of the 1D grid world following the MDP. $V^{*}$ represents the expected true value of each cell according to discount factor ($\gamma$) when the reward of the terminal state is one.
• Figure 1: The same principal component analysis (PCA) of the SR place field matrix learning history as shown in Figure 3, but drawn to the same scale. Except for the scale, the details are the same as in Figure 3.
• Figure 2: The simulated learning histories of the SR place field show that the SF agents, with the initial weight set with random weights, rapidly converges to the asymmetric SR place field. (A) Line plots of the learned SR place field of 50th cell after end of 10th, 50th, 100th, 300th episode are shown. Each line and shade show the averaged result with the standard deviation from 10 simulations in a grid world with 100 cells. Each row panel displays the SR agent (first row) or SF agents with different weight initialization methods (five rows below). (B) Rearranged line plots from A for comparing between the agents (SR, blue; SF weight initialization with identity matrix, orange; zero matrix, green; the Xavier method, red; the He method, purple; the uniform distribution, brown). Each row panel displays the simulated results after end of 10th, 25th, 50th, 100th, 300th, 500th episode. Note that the SF agents with random weights shows skewed SR place field at 50th episode, but other agents show symmetrical SR place field. (C) Learning histories of whole SR place field matrix in a grid world with 100 cells are shown according to episodes (column panels) and learning agents (row panels).
• Figure 2: L1 distances divided by the size of the matrix ($N \times N$) are shown. This normalization shows the distance between one element of the SR matrix. Random agents has shown great distance from the non-random agents.
• Figure 3: Principal component analysis (PCA) of SR place field matrix learning history shows that SF agents with random weights takes shorter route to converging optima. The simulated results from four different sizes of grid worlds (N=5, 25, 50, 100) are shown. Each dot shows PCA results of SR place fields after each episode (*, first episode). Each line shows historical route of SR place field learning from SR or SF agents (SR, blue; SF weight initialization with identity matrix, orange; zero matrix, green; the Xavier method, red; the He method, purple; the uniform distribution, brown). The average of the SR place field matrices from 10 simulations was used for PCA.