CosmoCore Affective Dream-Replay Reinforcement Learning for Code Generation

TL;DR

CosmoCore introduces an affective reinforcement learning framework for code generation by tagging model trajectories with valence $v\in[-1,1]$ and arousal $a\in[0,1]$, and replaying high-impact errors via a Dream Queue during off-policy updates. A lightweight affective tagger and a valence-gated buffer govern selective rehearsal and pruning, enabling more efficient learning than uniform replay. Across benchmarks such as HumanEval and BigCodeBench and in Mini-World PySpark simulations, CosmoCore reduces code hallucinations and speeds self-correction while boosting exploration for low-signal episodes. The work advances RLHF by incorporating emotion-like signals, offering practical improvements for robust, safer AI coding assistants and data pipelines, with code and a mini-world simulator released.

Abstract

We introduce CosmoCore, a neuroscience-inspired reinforcement learning (RL) architecture that integrates affective signals to enhance code generation in large language models (LLMs). Motivated by human and animal learning where embarrassment from mistakes drives rapid correction, as observed in training a puppy to avoid repeating errors after a single scolding CosmoCore tags code generation trajectories with valence and surprise using a lightweight multi-layer perceptron (MLP). High-negative valence (cringe) episodes, such as buggy code outputs, are prioritized in a Dream Queue for five-fold replay during off-policy updates, while low-surprise successes are pruned to prevent overconfidence and buffer bloat. Evaluated on code generation benchmarks like HumanEval and BigCodeBench, alongside simulations with a custom data pipeline environment, CosmoCore reduces hallucinated code (e.g., syntax errors or logical bugs) by 48\% and accelerates self-correction by 45\%. Local experiments using Hugging Face models in a PySpark environment validate these gains, with code snippets provided for replication. Ablations confirm valence tagging boosts curiosity in exploration, and pruning mitigates inefficiency. This framework extends RL from human feedback (RLHF) for more emotionally aware code assistants, with applications in IDEs and data pipelines. Code and the custom mini-world simulation are released.