Reducing Mental Workload through On-Demand Human Assistance for Physical Action Failures in LLM-based Multi-Robot Coordination

Abstract

Multi-robot coordination based on large language models (LLMs) has attracted growing attention, since LLMs enable the direct translation of natural language instructions into robot action plans by decomposing tasks and generating high-level plans. However, recovering from physical execution failures remains difficult, and tasks often stagnate due to the repetition of the same unsuccessful actions. While frameworks for remote robot operation using Mixed Reality were proposed, there have been few attempts to implement remote error resolution specifically for physical failures in multi-robot environments. In this study, we propose REPAIR (Robot Execution with Planned And Interactive Recovery), a human-in-the-loop framework that integrates remote error resolution into LLM-based multi-robot planning. In this method, robots execute tasks autonomously; however, when an irrecoverable failure occurs, the LLM requests assistance from an operator, enabling task continuity through remote intervention. Evaluations using a multi-robot trash collection task in a real-world environment confirmed that REPAIR significantly improves task progress (the number of items cleared within a time limit) compared to fully autonomous methods. Furthermore, for easily collectable items, it achieved task progress equivalent to full remote control. The results also suggested that the mental workload on the operator may differ in terms of physical demand and effort. The project website is https://emergentsystemlabstudent.github.io/REPAIR/.

Figures (5)

• Figure 2: An overview of our study. As multiple robots act based on planning generated by an LLM, they may fall into a loop state due to action failures. In such cases, the robots request assistance from an operator as needed, and the issue is resolved through remote error resolution.
• Figure 3: An overview of the proposed method. (a) The LLM takes a user's instruction, decomposes it based on robot capabilities and rules, and assigns tasks to each robot. (b) It then generates executable actions, which the robots perform; when failures occur, they request operator assistance. (c) The operator resolves the issue remotely and provides feedback, allowing the LLM to update its understanding and resume execution.
• Figure 4: Experimental Environment and Object Placements.
• Figure 5: Fig. \ref{['fig:result_nasa_tlx']}(a) shows the overall NASA-TLX score, while Fig. \ref{['fig:result_nasa_tlx']}(b)--(g) show the scores for each subscale. The gray lines in each figure represent changes in scores across conditions for the same subject. In Fig. \ref{['fig:result_nasa_tlx']}(b), (c), and (f), the Friedman test revealed significant differences among the three conditions ($p < 0.05$). A lower score indicates a better result (↓).
• Figure 6: Distribution of task progress and results of statistical comparisons for each condition. The higher the task progress, the better (↑).