Learning Transferable Skills in Action RPGs via Directed Skill Graphs and Selective Adaptation

TL;DR

The paper addresses lifelong learning in a challenging real-time control setting by factorizing combat control into five reusable skills arranged in a directed skill graph. It introduces a hierarchical curriculum (C→L→M→D→H) and per-skill objectives, enabling selective post-training where only downstream skills adapt during a phase shift from Phase 1 to Phase 2. Empirical results show substantial sample efficiency gains versus an end-to-end baseline and demonstrate that targeted fine-tuning of a small subset of skills recovers performance under limited interaction budgets. The findings argue that skill-graph curricula offer a practical pathway for scalable continual learning in complex, nonstationary environments.

Abstract

Lifelong agents should expand their competence over time without retraining from scratch or overwriting previously learned behaviors. We investigate this in a challenging real-time control setting (Dark Souls III) by representing combat as a directed skill graph and training its components in a hierarchical curriculum. The resulting agent decomposes control into five reusable skills: camera control, target lock-on, movement, dodging, and a heal-attack decision policy, each optimized for a narrow responsibility. This factorization improves sample efficiency by reducing the burden on any single policy and supports selective post-training: when the environment shifts from Phase 1 to Phase 2, only a subset of skills must be adapted, while upstream skills remain transferable. Empirically, we find that targeted fine-tuning of just two skills rapidly recovers performance under a limited interaction budget, suggesting that skill-graph curricula together with selective fine-tuning offer a practical pathway toward evolving, continually learning agents in complex real-time environments.

Figures (6)

• Figure 1: The Modular Skill Graph Architecture. At run-time, the global environment state (with dimension 25 in our experiments) is mapped into five specialized observation spaces ($o_t^k$), which are processed concurrently by independent skill policies ($\pi_k$) to generate action components. The grey arrows between policies denote the hierarchical training dependency: upstream skills (top) are trained first and held fixed, shaping the data distribution for the adaptive downstream skills (bottom).
• Figure 2: Return vs. training steps for $\mathcal{C}$, $\mathcal{L}$, and $\mathcal{M}$, averaged over five evaluation episodes at each 1k-step checkpoint. The shaded region denotes the 95% confidence interval.
• Figure 3: Return vs. training steps for $\mathcal{D}$ and $\mathcal{H}$, averaged over five evaluation episodes at each 1k-step checkpoint. The shaded region denotes the 95% confidence interval, and the dashed vertical line marks the Phase 1 $\rightarrow$ Phase 2 transition.
• Figure 4: Phase split used as a domain shift. Each subfigure shows a concatenation of two screenshots from the corresponding phase. The encounter was divided into Phase 1 and Phase 2 to evaluate transfer and selective fine-tuning under a controlled shift in boss behavior.
• Figure 5: End-to-end baseline struggles under the same state interface. The DQN baseline quickly plateaus, exhibiting little change in return or survival time beyond early training. The episode length plot shows that the agent couldn't even postpone early deaths even with its conservative strategy. The translucent line shows the raw value, and the darker line shows the 10-period moving average.