Heterogeneous Adversarial Play in Interactive Environments

TL;DR

Heterogeneous Adversarial Play (HAP) casts curriculum design as a minimax teacher–student game to autonomously generate and adapt task challenges across heterogeneous tasks. The framework uses a neural teacher that outputs a distribution over tasks $p_{\phi}(T|h_t)$ and a neural student that optimizes policy parameters $\theta$, formalized as $\min_{\phi}\max_{\theta} J(\theta,\phi)$ with the student maximizing $J_{student}(\theta)$ and the teacher minimizing $J_{teacher}(\phi)$. Key contributions include (i) a theoretical minimax formulation for asymmetric curriculum learning, (ii) empirical validation across Minigrid, CRAFT, and Crafter showing improved learning efficiency and robust curriculum generation, and (iii) human studies demonstrating curricula generated by the adversarial loop align with effective pedagogical principles. Extensions address cold-start warm-ups, entropy-based task selection, and safeguards against forgetting, while extended evaluations show applicability to supervised curricula and human learners. This work demonstrates that adversarial co-adaptation can autonomously discover instructional strategies that generalize toward both artificial agents and humans, reducing the need for handcrafted curricula.

Abstract

Self-play constitutes a fundamental paradigm for autonomous skill acquisition, whereby agents iteratively enhance their capabilities through self-directed environmental exploration. Conventional self-play frameworks exploit agent symmetry within zero-sum competitive settings, yet this approach proves inadequate for open-ended learning scenarios characterized by inherent asymmetry. Human pedagogical systems exemplify asymmetric instructional frameworks wherein educators systematically construct challenges calibrated to individual learners' developmental trajectories. The principal challenge resides in operationalizing these asymmetric, adaptive pedagogical mechanisms within artificial systems capable of autonomously synthesizing appropriate curricula without predetermined task hierarchies. Here we present Heterogeneous Adversarial Play (HAP), an adversarial Automatic Curriculum Learning framework that formalizes teacher-student interactions as a minimax optimization wherein task-generating instructor and problem-solving learner co-evolve through adversarial dynamics. In contrast to prevailing ACL methodologies that employ static curricula or unidirectional task selection mechanisms, HAP establishes a bidirectional feedback system wherein instructors continuously recalibrate task complexity in response to real-time learner performance metrics. Experimental validation across multi-task learning domains demonstrates that our framework achieves performance parity with SOTA baselines while generating curricula that enhance learning efficacy in both artificial agents and human subjects.

Figures (8)

• Figure 1: Comparison of different learning frameworks. (a) acl: A heuristic teacher selects tasks from a predefined curriculum pool and provides feedback to guide student learning through environments of increasing complexity. The teacher relies on domain expertise and rule-based policies to sequence tasks appropriately. (b) method: Our framework extends acl through adversarial co-evolution. The teacher learns to generate challenging but solvable tasks that maximize student learning, while the student adapts to solve the teacher's evolving problem proposals.
• Figure 2: Adversarial curriculum dynamics in a navigation benchmark. (a) Training reward curves demonstrate that method achieves faster convergence and higher overall performance compared to baseline approaches, reaching optimal performance around 35k steps. (b) Task-specific success rates reveal that method learns all difficulty levels uniformly, while baselines exhibit pronounced performance gaps between easier and harder tasks throughout training. (c) Underlying adversarial mechanism: solid lines show task success rates during evaluation, while dashed lines indicate corresponding task sampling probabilities from the teacher's perspective, illustrating the positive feedback loop (increased sampling for failed tasks) and negative feedback mechanism (reduced sampling for mastered tasks) that drive method's effectiveness.
• Figure 3: Task-specific performance breakdown in Crafter. Average scores across individual achievements calculated from task success rates. method performs comparably to DreamerV3 on most tasks while demonstrating superior performance on complex, multi-step challenges requiring hierarchical skill composition.
• Figure 4: Supervised curriculum learning performance on CIFAR-100 and RTE benchmarks.method consistently outperforms established curriculum learning baselines across both computer vision and NLP tasks, demonstrating particular robustness in challenging scenarios with class imbalance (CIFAR-100) and label noise (RTE). Results show superior convergence speed and final performance compared to data selection and loss reweighting approaches. Min-max ranges shown for method for visual clarity.
• Figure 5: Human learning performance across curriculum conditions in Minigrid. Comparison of human subjects' learning trajectories under three conditions: no tutorial, expert step-by-step tutorial, and method-generated tutorial, demonstrating the effectiveness of adversarial curriculum design for human learners.