Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Survival Dynamics of Neural and Programmatic Policies in Evolutionary Reinforcement Learning

Anton Roupassov-Ruiz, Yiyang Zuo

TL;DR

The paper addresses long-horizon robustness in evolutionary reinforcement learning by comparing neural policy (NERL) and programmatic policy (PERL) approaches within the Ackley-Littman ERL testbed. It introduces a fully specified, open-source reimplementation and a programmatic policy using soft differentiable decision lists (SDDL), evaluated with survival analysis tools including Kaplan-Meier curves and RMST. Empirically, PERL achieves a statistically significant survival advantage over NERL, with RMST higher by $201.69$ steps (PERL $=1277.14$, NERL $=1075.45$, $p\approx3.8\times10^{-75}$), and even learning-only PERL surpassing neural policies that combine learning and evolution. These results underscore the importance of policy representation, showing that structured, interpretable programmatic controllers can outperform neural networks in a complex, partially observable domain, while also enabling reproducibility and deeper behavioral interpretation.

Abstract

In evolutionary reinforcement learning tasks (ERL), agent policies are often encoded as small artificial neural networks (NERL). Such representations lack explicit modular structure, limiting behavioral interpretation. We investigate whether programmatic policies (PERL), implemented as soft, differentiable decision lists (SDDL), can match the performance of NERL. To support reproducible evaluation, we provide the first fully specified and open-source reimplementation of the classic 1992 Artificial Life (ALife) ERL testbed. We conduct a rigorous survival analysis across 4000 independent trials utilizing Kaplan-Meier curves and Restricted Mean Survival Time (RMST) metrics absent in the original study. We find a statistically significant difference in survival probability between PERL and NERL. PERL agents survive on average 201.69 steps longer than NERL agents. Moreover, SDDL agents using learning alone (no evolution) survive on average 73.67 steps longer than neural agents using both learning and evaluation. These results demonstrate that programmatic policies can exceed the survival performance of neural policies in ALife.

Survival Dynamics of Neural and Programmatic Policies in Evolutionary Reinforcement Learning

TL;DR

The paper addresses long-horizon robustness in evolutionary reinforcement learning by comparing neural policy (NERL) and programmatic policy (PERL) approaches within the Ackley-Littman ERL testbed. It introduces a fully specified, open-source reimplementation and a programmatic policy using soft differentiable decision lists (SDDL), evaluated with survival analysis tools including Kaplan-Meier curves and RMST. Empirically, PERL achieves a statistically significant survival advantage over NERL, with RMST higher by steps (PERL , NERL , ), and even learning-only PERL surpassing neural policies that combine learning and evolution. These results underscore the importance of policy representation, showing that structured, interpretable programmatic controllers can outperform neural networks in a complex, partially observable domain, while also enabling reproducibility and deeper behavioral interpretation.

Abstract

In evolutionary reinforcement learning tasks (ERL), agent policies are often encoded as small artificial neural networks (NERL). Such representations lack explicit modular structure, limiting behavioral interpretation. We investigate whether programmatic policies (PERL), implemented as soft, differentiable decision lists (SDDL), can match the performance of NERL. To support reproducible evaluation, we provide the first fully specified and open-source reimplementation of the classic 1992 Artificial Life (ALife) ERL testbed. We conduct a rigorous survival analysis across 4000 independent trials utilizing Kaplan-Meier curves and Restricted Mean Survival Time (RMST) metrics absent in the original study. We find a statistically significant difference in survival probability between PERL and NERL. PERL agents survive on average 201.69 steps longer than NERL agents. Moreover, SDDL agents using learning alone (no evolution) survive on average 73.67 steps longer than neural agents using both learning and evaluation. These results demonstrate that programmatic policies can exceed the survival performance of neural policies in ALife.
Paper Structure (35 sections, 17 equations, 2 figures, 3 tables)

This paper contains 35 sections, 17 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Between-Policy Comparison.
  4. Evaluation Objective.
  5. Related Work
  6. ERL and the Ackley–Littman Testbed
  7. Limited Interpretability of Neural Policies in ERL
  8. Lack of Quantitative Evaluation
  9. Limited Reproducibility
  10. Proposed Approach
  11. Input
  12. Input Vector
  13. NERL
  14. PERL
  15. Action Network
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Overview of NERL. Reproduced from work by ackley1991interactions
  • Figure 2: Kaplan–Meier survival curves comparing all agent strategies.