Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Code Evolution for Control: Synthesizing Policies via LLM-Driven Evolutionary Search

Ping Guo, Chao Li, Yinglan Feng, Chaoning Zhang

TL;DR

The paper addresses the challenge of designing trustworthy, interpretable control policies for autonomous systems, proposing to synthesize policies as executable Python code via LLM-driven evolutionary search. By treating policy design as code evolution and using three LLM-based mutation/crossover strategies (FunSearch, EoH, EvoEngineer), the authors demonstrate that rich contextual feedback enables substantially better performance and interpretability than traditional neural policies. On LunarLander-v3, EvoEngineer outperforms simpler LLM strategies and matches or exceeds PPO in certain metrics, with EvoEngineer+ delivering a highly interpretable 59-line policy that achieves a 70% success rate, albeit with some trade-offs in average reward. The results highlight the potential of combining foundation models with evolutionary computation to synthesize trustworthy, verifiable control policies, particularly in safety-critical settings where interpretability and auditability are crucial.

Abstract

Designing effective control policies for autonomous systems remains a fundamental challenge, traditionally addressed through reinforcement learning or manual engineering. While reinforcement learning has achieved remarkable success, it often suffers from high sample complexity, reward shaping difficulties, and produces opaque neural network policies that are hard to interpret or verify. Manual design, on the other hand, requires substantial domain expertise and struggles to scale across diverse tasks. In this work, we demonstrate that LLM-driven evolutionary search can effectively synthesize interpretable control policies in the form of executable code. By treating policy synthesis as a code evolution problem, we harness the LLM's prior knowledge of programming patterns and control heuristics while employing evolutionary search to explore the solution space systematically. We implement our approach using EvoToolkit, a framework that seamlessly integrates LLM-driven evolution with customizable fitness evaluation. Our method iteratively evolves populations of candidate policy programs, evaluating them against task-specific objectives and selecting superior individuals for reproduction. This process yields compact, human-readable control policies that can be directly inspected, modified, and formally verified. This work highlights the potential of combining foundation models with evolutionary computation for synthesizing trustworthy control policies in autonomous systems. Code is available at https://github.com/pgg3/EvoControl.

Code Evolution for Control: Synthesizing Policies via LLM-Driven Evolutionary Search

TL;DR

The paper addresses the challenge of designing trustworthy, interpretable control policies for autonomous systems, proposing to synthesize policies as executable Python code via LLM-driven evolutionary search. By treating policy design as code evolution and using three LLM-based mutation/crossover strategies (FunSearch, EoH, EvoEngineer), the authors demonstrate that rich contextual feedback enables substantially better performance and interpretability than traditional neural policies. On LunarLander-v3, EvoEngineer outperforms simpler LLM strategies and matches or exceeds PPO in certain metrics, with EvoEngineer+ delivering a highly interpretable 59-line policy that achieves a 70% success rate, albeit with some trade-offs in average reward. The results highlight the potential of combining foundation models with evolutionary computation to synthesize trustworthy, verifiable control policies, particularly in safety-critical settings where interpretability and auditability are crucial.

Abstract

Designing effective control policies for autonomous systems remains a fundamental challenge, traditionally addressed through reinforcement learning or manual engineering. While reinforcement learning has achieved remarkable success, it often suffers from high sample complexity, reward shaping difficulties, and produces opaque neural network policies that are hard to interpret or verify. Manual design, on the other hand, requires substantial domain expertise and struggles to scale across diverse tasks. In this work, we demonstrate that LLM-driven evolutionary search can effectively synthesize interpretable control policies in the form of executable code. By treating policy synthesis as a code evolution problem, we harness the LLM's prior knowledge of programming patterns and control heuristics while employing evolutionary search to explore the solution space systematically. We implement our approach using EvoToolkit, a framework that seamlessly integrates LLM-driven evolution with customizable fitness evaluation. Our method iteratively evolves populations of candidate policy programs, evaluating them against task-specific objectives and selecting superior individuals for reproduction. This process yields compact, human-readable control policies that can be directly inspected, modified, and formally verified. This work highlights the potential of combining foundation models with evolutionary computation for synthesizing trustworthy control policies in autonomous systems. Code is available at https://github.com/pgg3/EvoControl.
Paper Structure (22 sections, 2 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 22 sections, 2 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Control
  4. Program Synthesis and Neuro-Symbolic Methods
  5. LLM-Driven Code Evolution
  6. Methodology
  7. Problem Formulation
  8. LLM-Driven Evolution Framework
  9. Fitness Evaluation
  10. Experiments
  11. Experimental Setup
  12. Results
  13. Main Results
  14. Convergence Analysis
  15. Sample Efficiency Analysis
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Best fitness vs. generation for EoH and EvoEngineer. Shaded regions indicate standard deviation across 5 seeds.
  • Figure 2: Best evolved policy (simplified excerpt). The full policy contains additional edge-case handling for angular velocity and leg contact states.