Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

Zhishen Sun, Sizhe Dang, Guang Dai, Haishan Ye

TL;DR

ESSAM tackles the memory bottleneck in RL fine-tuning of LLMs by uniting Evolution Strategies with Sharpness-Aware Maximization in a zero-order, full-parameter framework. Through a two-stage, SAM-informed update and memory-saving procedures, ESSAM steers optimization toward flatter minima while avoiding backpropagation, achieving GSM8K performance comparable to PPO and GRPO while using inference-level GPU memory. Empirical results on Qwen and LLaMA models show ESSAM not only closes the gap with RL methods but also delivers substantial memory reductions (approximately 18x vs PPO and 10x vs GRPO). These findings suggest ESSAM as a practical, resource-efficient approach for math reasoning fine-tuning in settings with constrained compute, with strong potential for open-source deployment.

Abstract

Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes it hard to use in settings with limited resources. To reduce these issues, we propose Evolution Strategies with Sharpness-Aware Maximization (ESSAM), a full parameter fine-tuning framework that tightly combines the zero-order search in parameter space from Evolution Strategies (ES) with the Sharpness-Aware Maximization (SAM) to improve generalization. We conduct fine-tuning experiments on the mainstream mathematica reasoning task GSM8K. The results show that ESSAM achieves an average accuracy of 78.27\% across all models and its overall performance is comparable to RL methods. It surpasses classic RL algorithm PPO with an accuracy of 77.72\% and is comparable to GRPO with an accuracy of 78.34\%, and even surpassing them on some models. In terms of GPU memory usage, ESSAM reduces the average GPU memory usage by $18\times$ compared to PPO and by $10\times$ compared to GRPO, achieving an extremely low GPU memory usage.

ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

TL;DR

ESSAM tackles the memory bottleneck in RL fine-tuning of LLMs by uniting Evolution Strategies with Sharpness-Aware Maximization in a zero-order, full-parameter framework. Through a two-stage, SAM-informed update and memory-saving procedures, ESSAM steers optimization toward flatter minima while avoiding backpropagation, achieving GSM8K performance comparable to PPO and GRPO while using inference-level GPU memory. Empirical results on Qwen and LLaMA models show ESSAM not only closes the gap with RL methods but also delivers substantial memory reductions (approximately 18x vs PPO and 10x vs GRPO). These findings suggest ESSAM as a practical, resource-efficient approach for math reasoning fine-tuning in settings with constrained compute, with strong potential for open-source deployment.

Abstract

Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes it hard to use in settings with limited resources. To reduce these issues, we propose Evolution Strategies with Sharpness-Aware Maximization (ESSAM), a full parameter fine-tuning framework that tightly combines the zero-order search in parameter space from Evolution Strategies (ES) with the Sharpness-Aware Maximization (SAM) to improve generalization. We conduct fine-tuning experiments on the mainstream mathematica reasoning task GSM8K. The results show that ESSAM achieves an average accuracy of 78.27\% across all models and its overall performance is comparable to RL methods. It surpasses classic RL algorithm PPO with an accuracy of 77.72\% and is comparable to GRPO with an accuracy of 78.34\%, and even surpassing them on some models. In terms of GPU memory usage, ESSAM reduces the average GPU memory usage by compared to PPO and by compared to GRPO, achieving an extremely low GPU memory usage.
Paper Structure (20 sections, 3 theorems, 74 equations, 8 figures, 5 tables, 4 algorithms)

This paper contains 20 sections, 3 theorems, 74 equations, 8 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Evolution Strategies with Sharpness-Aware Maximization
  5. Memory Saving Mechanism
  6. Understanding the algorithm design of ESSAM
  7. Experiment
  8. Experimental Setup
  9. Experimental Results
  10. Main Results
  11. GPU Memory Usage and Time Efficiency Analysis
  12. Conclusion
  13. Evolution Strategies
  14. Proof
  15. Proof of Proposition \ref{['pro1']}
  16. ...and 5 more sections

Key Result

Proposition 3.1

Let the stochastic gradient estimation $g_t$ defined in Eq. eq:grad_est and the variance $s_r^2$ defined in Eq. eq:10. Then it holds that where d denotes the number of model parameters.

Figures (8)

  • Figure 1: An illustration of the ESSAM parameter update.
  • Figure 2: The average accuracy of each algorithm on all models for the GSM8K task (%).
  • Figure 3: An example GSM8K problem and the prompt template.
  • Figure 4: The training mean reward curves of ESSAM and ES. These curves show that ESSAM has better training trend and converges earlier than ES, leading to better computational efficiency. More results are presented in the Appendix \ref{['D']}.
  • Figure 5: GPU memory usage when fine-tuning different LLMs with different algorithms. More details can be found in Appendix \ref{['B']}.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Proposition 3.1
  • Lemma 2.1
  • proof
  • Lemma 2.2
  • proof
  • proof