HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym

TL;DR

HDDLgym addresses the gap between hierarchical planning and reinforcement learning by automatically converting HDDL domains and problems into OpenAI Gym environments, enabling RL policies to guide hierarchical planning in both single- and multi-agent settings. The framework introduces an agent-centric HDDL extension, a planner that uses RL outputs to assemble action hierarchies, and a flexible observation/action encoding that supports centralized or decentralized planning. Key contributions include the HDDLGym tool, a protocol for extending HDDL domains to multi-agent configurations, and demonstrations in the Transport and Overcooked domains with thorough evaluation and visualization facilities. This approach enables systematic study of RL within hierarchical planning, with practical impact for multi-agent coordination and complex, long-horizon tasks.

Abstract

In recent years, reinforcement learning (RL) methods have been widely tested using tools like OpenAI Gym, though many tasks in these environments could also benefit from hierarchical planning. However, there is a lack of a tool that enables seamless integration of hierarchical planning with RL. Hierarchical Domain Definition Language (HDDL), used in classical planning, introduces a structured approach well-suited for model-based RL to address this gap. To bridge this integration, we introduce HDDLGym, a Python-based tool that automatically generates OpenAI Gym environments from HDDL domains and problems. HDDLGym serves as a link between RL and hierarchical planning, supporting multi-agent scenarios and enabling collaborative planning among agents. This paper provides an overview of HDDLGym's design and implementation, highlighting the challenges and design choices involved in integrating HDDL with the Gym interface, and applying RL policies to support hierarchical planning. We also provide detailed instructions and demonstrations for using the HDDLGym framework, including how to work with existing HDDL domains and problems from International Planning Competitions, exemplified by the Transport domain. Additionally, we offer guidance on creating new HDDL domains for multi-agent scenarios and demonstrate the practical use of HDDLGym in the Overcooked domain. By leveraging the advantages of HDDL and Gym, HDDLGym aims to be a valuable tool for studying RL in hierarchical planning, particularly in multi-agent contexts.

Figures (4)

• Figure 1: Examples of the Transport and Overcooked environments in HDDLGym
• Figure 2: HDDLGym high-level architecture. Outputs of RL policy help HDDLGym Planner update the action hierarchy of each agent. Then, primitive actions are extracted from the hierarchies, and applied to the environment.
• Figure 3: HTNs of the Overcooked domain.
• Figure 4: Training Dynamics Analysis. Figures A, B, and C show the training losses for the Transport domain with 1, 2, and 3 agents, highlighting longer convergence times as the number of agents increases. Figures D and E display the PPO policy's training progression for the 1-agent Transport problem, including cumulative reward, success rate, planning time, and steps.

Theorems & Definitions (2)

• Definition 1
• Definition 2