Low Rank Factorizations are Indirect Encodings for Deep Neuroevolution

TL;DR

Low-rank factorizations are proposed as an indirect encoding for neuroevolution to constrain the search space of weights while injecting structure across the network. The method represents each weight matrix as a product of low-rank factors and evolves the factors with a seed-based scalable GA, comparing against non-factorized baselines and a small-parameter control. Across transformer language modeling, CarRacing, and Atari tasks, factorized neuroevolution is competitive with or outperforms baselines, with embedding-factorized layers driving early gains in language tasks. The findings suggest that leveraging low intrinsic dimensionality and seed-based updates can yield scalable, gradient-free optimization that benefits from insights of backpropagation.

Abstract

Deep neuroevolution is a highly scalable alternative to reinforcement learning due to its unique ability to encode network updates in a small number of bytes. Recent insights from traditional deep learning indicate high-dimensional models possess intrinsic, low-rank structure. In this work, we introduce low-rank, factorized neuroevolution: an indirect encoding through which we can search a small space of low-rank factors that enforce underlying structure across a network's weights. We compare our approach with non-factorized networks of similar and smaller size to understand how much performance can be attributed to the smaller search space. We evaluate our method on a language modeling task using transformers, as well as continuous and discrete vision-based reinforcement learning tasks. Our study shows that low-rank, factorized neuroevolution outperforms or is competitive with non-factorized neuroevolution, performing notably well on language modeling. Our results also suggest deleterious factorized mutations have a stronger negative impact on performance than deleterious non-factorized mutations, which significantly reduces the runtime on environments with early termination for bad performers. More broadly, these results show how we can use insights from backpropgation-based methods to enhance neuroevolution

Figures (7)

• Figure 1: Representations for an arbitrary weight matrix in each representation type. Green matrices represent the genotype, i.e., the parameters we directly mutate, whereas blue matrices represent the phenotype, the final, developed set of parameters which are actually used.
• Figure 2: Training loss of best individual on the first 1024 sequences of the TinyStories Dataset, averaged over 7 trials. 95% confidence intervals are shown, but small.
• Figure 3: Training loss of best individual on the first 1024 sequences of the TinyStories dataset across three ranks, averaged over 7 trials. The smallest setting has an $k_\text{embed}$=4, $k_\text{other}$=1; the medium has $k_\text{embed}$=32, $k_\text{other}$=4; large has $k_\text{embed}$=64, $k_\text{other}$=16.
• Figure 4: CarRacing Environment. Agents must learn to quickly drive a lap around a randomly generated track, given $96\times96\times3$ RGB observations, which we resize to $64\times64\times3$.
• Figure 5: Score of best individual per generation on CarRacing environment over fifteen experiments, 95% confidence intervals are shown.