Generative World Models of Tasks: LLM-Driven Hierarchical Scaffolding for Embodied Agents
Brennen Hill
TL;DR
End-to-end MARL struggles with task sparsity in long-horizon, multi-agent tasks such as robotic soccer. The paper advocates Hierarchical Task Environments (HTEs) and dynamic LLM-driven scaffolding to externalize task structure, guiding exploration and enabling hierarchical reasoning. It synthesizes a trend in 2024 soccer research toward hybrid symbolic-hierarchical methods (e.g., HTN and BSN) and outlines concrete HTE components, including intrinsic rewards, procedural curricula, and LLM-based plan generation with verification. This approach promises improved sample efficiency, better generalization through composition, and stronger Sim2Real transfer by decomposing problems into manageable sub-tasks rather than indiscriminately scaling data.
Abstract
Recent advances in agent development have focused on scaling model size and raw interaction data, mirroring successes in large language models. However, for complex, long-horizon multi-agent tasks such as robotic soccer, this end-to-end approach often fails due to intractable exploration spaces and sparse rewards. We propose that an effective world model for decision-making must model the world's physics and also its task semantics. A systematic review of 2024 research in low-resource multi-agent soccer reveals a clear trend towards integrating symbolic and hierarchical methods, such as Hierarchical Task Networks (HTNs) and Bayesian Strategy Networks (BSNs), with multi-agent reinforcement learning (MARL). These methods decompose complex goals into manageable subgoals, creating an intrinsic curriculum that shapes agent learning. We formalize this trend into a framework for Hierarchical Task Environments (HTEs), which are essential for bridging the gap between simple, reactive behaviors and sophisticated, strategic team play. Our framework incorporates the use of Large Language Models (LLMs) as generative world models of tasks, capable of dynamically generating this scaffolding. We argue that HTEs provide a mechanism to guide exploration, generate meaningful learning signals, and train agents to internalize hierarchical structure, enabling the development of more capable and general-purpose agents with greater sample efficiency than purely end-to-end approaches.