Evolutionary Strategies lead to Catastrophic Forgetting in LLMs

TL;DR

The paper investigates the viability of Evolutionary Strategies (ES) for continual learning in large language models (LLMs) and compares it to gradient-based GRPO, highlighting memory-efficiency advantages but potential instability. Through comprehensive experiments on Countdown, GSM8K, MATH, and OlympiadBench across Qwen-2.5-1.5B-Instruct and Llama-3.2-1B-Instruct, ES achieves competitive performance but does not consistently outperform GRPO and exhibits pronounced forgetting during extended fine-tuning. The analysis reveals that ES updates are broader with significantly larger $\ell_2$ norms than GRPO, causing global parameter drift and interference with prior skills. The findings underscore a trade-off between memory efficiency and forgetting in gradient-free continual learning and provide a public codebase to spur future mitigation strategies.

Abstract

One of the biggest missing capabilities in current AI systems is the ability to learn continuously after deployment. Implementing such continually learning systems have several challenges, one of which is the large memory requirement of gradient-based algorithms that are used to train state-of-the-art LLMs. Evolutionary Strategies (ES) have recently re-emerged as a gradient-free alternative to traditional learning algorithms and have shown encouraging performance on specific tasks in LLMs. In this paper, we perform a comprehensive analysis of ES and specifically evaluate its forgetting curves when training for an increasing number of update steps. We first find that ES is able to reach performance numbers close to GRPO for math and reasoning tasks with a comparable compute budget. However, and most importantly for continual learning, the performance gains in ES is accompanied by significant forgetting of prior abilities, limiting its applicability for training models online. We also explore the reason behind this behavior and show that the updates made using ES are much less sparse and have orders of magnitude larger $\ell_2$ norm compared to corresponding GRPO updates, explaining the contrasting forgetting curves between the two algorithms. With this study, we aim to highlight the issue of forgetting in gradient-free algorithms like ES and hope to inspire future work to mitigate these issues.

Figures (7)

• Figure 1: Pareto front between new task (Countdown) and old task (HellaSwag) performance across fine-tuning with ES and GRPO.
• Figure 2: Prior task accuracy (%; HellaSwag) vs. training iteration for ES and GRPO fine-tuning. GRPO-trained models exhibit stable prior task accuracy across training iteration, while ES-trained models degrade with continued fine-tuning.
• Figure 3: Relationship between Frobenius norm of a model update and number of training iterations on a new task. ES-trained models drift several orders of magnitude more than GRPO-trained models.
• Figure 4: Layerwise sparsity of parameter updates during fine-tuning (higher indicates more sparse updates). ES exhibits broadly distributed, dense updates across components, whereas GRPO updates are overall highly sparse across LLM parameter groups, consistent with more targeted parameter changes.
• Figure A1: Relationship between KL divergence and task performance. Top row: new-task accuracy (Countdown). Bottom row: prior-task accuracy (HellaSwag). Training step indicated per sample. ES exhibits increasing KL accompanied by degradation on the prior task, whereas GRPO maintains stable performance across a broader KL range.