Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

Jennifer Dodgson, Alfath Daryl Alhajir, Michael Joedhitya, Akira Rafhael Janson Pattirane, Surender Suresh Kumar, Joseph Lim, C. H. Peh, Adith Ramdas, Steven Zhang Zhexu

TL;DR

This paper tackles the instability of self-training by introducing environment-mediated selection and negative-space learning (NSL), removing reliance on external reward shaping or curated data. It demonstrates a resource-constrained sandbox where only actions that persistently improve environmental state influence future training, enabling open-ended improvement without human guidance. Across three lineages (Terese, Miri, Katalin), the study shows that memory-bounded self-training can yield sustainable gains, with Miri achieving monotonic progress under limited memory while Katalin reveals risks of instability from history-based selection. The findings suggest a pathway toward robust, generalisable autonomous systems that learn through real-world constraints rather than proxy rewards, with implications for continuous self-improvement and meta-learning in non-stationary environments.

Abstract

Self-training systems often degenerate due to the lack of an external criterion for judging data quality, leading to reward hacking and semantic drift. This paper provides a proof-of-concept system architecture for stable self-training under sparse external feedback and bounded memory, and empirically characterises its learning dynamics and failure modes. We introduce a self-training architecture in which learning is mediated exclusively by environmental viability, rather than by reward, objective functions, or externally defined fitness criteria. Candidate behaviours are executed under real resource constraints, and only those whose environmental effects both persist and preserve the possibility of future interaction are propagated. The environment does not provide semantic feedback, dense rewards, or task-specific supervision; selection operates solely through differential survival of behaviours as world-altering events, making proxy optimisation impossible and rendering reward-hacking evolutionarily unstable. Analysis of semantic dynamics shows that improvement arises primarily through the persistence of effective and repeatable strategies under a regime of consolidation and pruning, a paradigm we refer to as negative-space learning (NSL), and that models develop meta-learning strategies (such as deliberate experimental failure in order to elicit informative error messages) without explicit instruction. This work establishes that environment-grounded selection enables sustainable open-ended self-improvement, offering a viable path toward more robust and generalisable autonomous systems without reliance on human-curated data or complex reward shaping.

Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

TL;DR

This paper tackles the instability of self-training by introducing environment-mediated selection and negative-space learning (NSL), removing reliance on external reward shaping or curated data. It demonstrates a resource-constrained sandbox where only actions that persistently improve environmental state influence future training, enabling open-ended improvement without human guidance. Across three lineages (Terese, Miri, Katalin), the study shows that memory-bounded self-training can yield sustainable gains, with Miri achieving monotonic progress under limited memory while Katalin reveals risks of instability from history-based selection. The findings suggest a pathway toward robust, generalisable autonomous systems that learn through real-world constraints rather than proxy rewards, with implications for continuous self-improvement and meta-learning in non-stationary environments.

Abstract

Self-training systems often degenerate due to the lack of an external criterion for judging data quality, leading to reward hacking and semantic drift. This paper provides a proof-of-concept system architecture for stable self-training under sparse external feedback and bounded memory, and empirically characterises its learning dynamics and failure modes. We introduce a self-training architecture in which learning is mediated exclusively by environmental viability, rather than by reward, objective functions, or externally defined fitness criteria. Candidate behaviours are executed under real resource constraints, and only those whose environmental effects both persist and preserve the possibility of future interaction are propagated. The environment does not provide semantic feedback, dense rewards, or task-specific supervision; selection operates solely through differential survival of behaviours as world-altering events, making proxy optimisation impossible and rendering reward-hacking evolutionarily unstable. Analysis of semantic dynamics shows that improvement arises primarily through the persistence of effective and repeatable strategies under a regime of consolidation and pruning, a paradigm we refer to as negative-space learning (NSL), and that models develop meta-learning strategies (such as deliberate experimental failure in order to elicit informative error messages) without explicit instruction. This work establishes that environment-grounded selection enables sustainable open-ended self-improvement, offering a viable path toward more robust and generalisable autonomous systems without reliance on human-curated data or complex reward shaping.
Paper Structure (31 sections, 11 equations, 23 figures, 11 tables)

This paper contains 31 sections, 11 equations, 23 figures, 11 tables.

Table of Contents

  1. Abstract
  2. Problem: Endogenous Selection in Self-Training
  3. Design Requirement: External, Consequence-Based Selection
  4. Environment Construction: Resource-Limited Execution Contexts
  5. Agent–Environment Interaction Loop
  6. Prompt Structure
  7. Training Pipeline
  8. Sandbox Design
  9. Result Data: Continued Learning With and Without Data-Scaling
  10. Terese lineage:
  11. Miri lineage:
  12. Katalin lineage:
  13. Terese Fine-Tuning (all generated datasets)
  14. Miri Fine-Tuning (Three Most Recent Datasets)
  15. Katalin Fine-Tuning (Three Highest-Space- Freed Datasets)
  16. ...and 16 more sections

Figures (23)

  • Figure 1: Simplified process diagram.
  • Figure 2: Chaining LoRAs to achieve incremental fine tuning without catastrophic forgetting
  • Figure 3: Space taken over per iteration as a % of total space available. Note that we use 68% confidence intervals here to improve visual resolution of temporal trends; 95% confidence intervals are reported above.
  • Figure 4: Average space taken over per run (MB). Note that we use 68% confidence intervals here to improve visual resolution of temporal trends; 95% confidence intervals are reported above.
  • Figure 5: Cumulative composite improvement scores. Note that we use 68% confidence intervals here to improve visual resolution of temporal trends; 95% confidence intervals are reported above.
  • ...and 18 more figures