Evolving Programmatic Skill Networks

TL;DR

The paper addresses continual skill acquisition in open-ended embodied environments by representing skills as executable symbolic programs that form a growing, compositional directed graph $\mathcal{N}_t$. It introduces three LLM-enabled mechanisms—Reflect for trace-based fault localization, maturity-aware update gating for stability with plasticity, and canonical structural refactoring with rollback validation—that drive online learning and network consolidation, with learning dynamics expressed through $V(s)$, $P(\text{update } s)$, and a composite objective $J(\mathcal{N})$. The authors show that PSN’s learning dynamics resemble neural-network training in structure, operating across fast, intermediate, and slow timescales, and demonstrate robust skill reuse, rapid adaptation, and strong generalization on MineDojo (Minecraft) and Crafter benchmarks. Overall, PSN provides a principled, modular approach to continual, scalable learning that bridges symbolic program synthesis and data-driven planning, with potential impact on scalable, reusable AI systems.

Abstract

We study continual skill acquisition in open-ended embodied environments where an agent must construct, refine, and reuse an expanding library of executable skills. We introduce the Programmatic Skill Network (PSN), a framework in which skills are executable symbolic programs forming a compositional network that evolves through experience. PSN defines three core mechanisms instantiated via large language models: (1)REFLECT for structured fault localization over skill compositions, (2) progressive optimization with maturity-aware update gating that stabilizes reliable skills while maintaining plasticity for uncertain ones, and (3) canonical structural refactoring under rollback validation that maintains network compactness. We further show that PSN's learning dynamics exhibit structural parallels to neural network training. Experiments on MineDojo and Crafter demonstrate robust skill reuse, rapid adaptation, and strong generalization across open-ended task distributions.\footnote{We plan to open-source the code.

Figures (13)

• Figure 1: The Programmatic Skill Network (PSN) framework. The agent maintains a skill network $\mathcal{N}_t$ where the hybrid planner selects or synthesizes skills; the PSN manager executes them. On failure, the skill optimizer performs trace-based credit assignment; on success, the online refactor restructures the network. This induces learning dynamics analogous to neural network training: fault localization as backpropagation, maturity gating as learning rate scheduling, and refactoring as architecture search.
• Figure 2: Tech tree mastery on Minecraft.
• Figure 3: Cumulative Reward on Crafter. Shorter curves indicate earlier agent death due to Crafter's survival mechanics (hostile mobs, hunger, hazards).
• Figure 4: Skill Retention Rate under continual learning setting on Minecraft. PSN consistently preserves previously mastered skills, while Voyager exhibits severe catastrophic forgetting as training progresses.
• Figure 5: The cumulative success rate of tasks for PSN w/ and w/o maturity gating, on Minecraft.