Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation
Shaojun Xu, Xusheng Luo, Yutong Huang, Letian Leng, Ruixuan Liu, Changliu Liu
TL;DR
This work tackles the problem of instructing multiple robots with long-horizon tasks expressed in natural language. It introduces Nl2Hltl2Plan, a two-stage neuro-symbolic pipeline that first extracts a Hierarchical Task Tree (HTT) from language and then translates subtasks into hierarchical HLTL (a hierarchical, syntactically co-safe LTL) suitable for off-the-shelf planners. By grounding language in hierarchical temporal logic and using HTTs to maintain task structure, the approach achieves higher success rates and lower planning costs in both simulated AI2-THOR scenarios and real-world tabletop setups, including multi-robot handovers. The results demonstrate the viability and practicality of using HLTL as an interpretable, scalable intermediate representation for multi-robot planning driven by natural language, with clear pathways for improving robustness via syntax/semantic checking and HTT reorganization. This work advances the integration of LLMs with formal planning for multi-robot systems, enabling more complex, user-friendly, and efficient task execution in real-world environments.
Abstract
To enable non-experts to specify long-horizon, multi-robot collaborative tasks, language models are increasingly used to translate natural language commands into formal specifications. However, because translation can occur in multiple ways, such translations may lack accuracy or lead to inefficient multi-robot planning. Our key insight is that concise hierarchical specifications can simplify planning while remaining straightforward to derive from human instructions. We propose Nl2Hltl2Plan, a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL) and solves the corresponding planning problem. The translation involves two steps leveraging Large Language Models (LLMs). First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations. Next, a fine-tuned LLM converts sub-tasks into flat LTL formulas, which are aggregated into hierarchical specifications, with the lowest level corresponding to ordered robot actions. These specifications are then used with off-the-shelf planners. Our Nl2Hltl2Plan demonstrates the potential of LLMs in hierarchical reasoning for multi-robot task planning. Evaluations in simulation and real-world experiments with human participants show that Nl2Hltl2Plan outperforms existing methods, handling more complex instructions while achieving higher success rates and lower costs in task allocation and planning. Additional details are available at https://nl2hltl2plan.github.io .