When Does Neuroevolution Outcompete Reinforcement Learning in Transfer Learning Tasks?
Eleni Nisioti, Joachim Winther Pedersen, Erwan Plantec, Milton L. Montero, Sebastian Risi
TL;DR
The paper investigates when neuroevolution (NE) can outperform reinforcement learning (RL) in transfer learning by introducing two curricula-based benchmarks, Stepping gates and Ecorobot, and evaluating a spectrum of NE and RL methods. It systematically compares direct (NEAT) and indirect (HyperNEAT) encodings, as well as diversity-driven (MAP-Elites) and gradient-free optimizers (CMA-ES) against PPO baselines, revealing that direct encodings generally transfer better across tasks, with NEAT often matching or surpassing CMA-ES, while indirect encodings excel at escaping local optima but struggle with skill transfer. The results highlight that curriculum structure and task complexity, especially with evolving morphologies, critically shape transfer performance, and that no single method yet solves both high-level transfer and complex locomotion. The study suggests hybrid encoding strategies that combine the strengths of direct and indirect mappings and calls for scalable benchmarks to push NE toward real-world applicability.
Abstract
The ability to continuously and efficiently transfer skills across tasks is a hallmark of biological intelligence and a long-standing goal in artificial systems. Reinforcement learning (RL), a dominant paradigm for learning in high-dimensional control tasks, is known to suffer from brittleness to task variations and catastrophic forgetting. Neuroevolution (NE) has recently gained attention for its robustness, scalability, and capacity to escape local optima. In this paper, we investigate an understudied dimension of NE: its transfer learning capabilities. To this end, we introduce two benchmarks: a) in stepping gates, neural networks are tasked with emulating logic circuits, with designs that emphasize modular repetition and variation b) ecorobot extends the Brax physics engine with objects such as walls and obstacles and the ability to easily switch between different robotic morphologies. Crucial in both benchmarks is the presence of a curriculum that enables evaluating skill transfer across tasks of increasing complexity. Our empirical analysis shows that NE methods vary in their transfer abilities and frequently outperform RL baselines. Our findings support the potential of NE as a foundation for building more adaptable agents and highlight future challenges for scaling NE to complex, real-world problems.