InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy

TL;DR

InteLiPlan tackles the challenge of robust domestic-robot autonomy by combining a lightweight, onboard LLM-based planner with multimodal perception and reachability-based feasibility checks. A human-in-the-loop supports real-time intervention for failures or ambiguities, enabling effective replanning without offboard computation. By using PEFT with LoRA on a compact, textualized multimodal dataset, the approach remains robot-agnostic and capable of onboard deployment, achieving high success in fetch tasks and competitive performance against state-of-the-art planners. The system demonstrates cross-embodiment transfer on HSR and Anymal D with a Z1 arm and delivers sub-7-second planning, indicating practical viability for real-world domestic scenarios. Limitations include the absence of motion-level replanning, which the authors suggest addressing via a dual-process integration with reactive low-level control in future work.

Abstract

We introduce an interactive LLM-based framework designed to enhance the autonomy and robustness of domestic robots, targeting embodied intelligence. Our approach reduces reliance on large-scale data and incorporates a robot-agnostic pipeline that embodies an LLM. Our framework, InteLiPlan, ensures that the LLM's decision-making capabilities are effectively aligned with robotic functions, enhancing operational robustness and adaptability, while our human-in-the-loop mechanism allows for real-time human intervention when user instruction is required. We evaluate our method in both simulation and on the real robot platforms, including a Toyota Human Support Robot and an ANYmal D robot with a Unitree Z1 arm. Our method achieves a 95% success rate in the `fetch me' task completion with failure recovery, highlighting its capability in both failure reasoning and task planning. InteLiPlan achieves comparable performance to state-of-the-art LLM-based robotics planners, while using only real-time onboard computing. Project website: https://kimtienly.github.io/InteLiPlan.

Figures (6)

• Figure 1: Step-by-step execution of InteLiPlan result on the physical Anymal D-Unitree Z1. In our system, the robot receives either requests or guidance from humans (< user>). Our lightweight onboard planner will generate the robot actions (< robot>) considering the success of targeted object detection (< vision>), whole-body feasibility score (< feasibility>) and execute them sequentially.
• Figure 2: System overview. Our multimodal planner integrates the user’s textual command/intervention, visual perception, and action feasibility score as inputs to a fine-tuned lightweight LLM. The LLM will then generate an action sequence for a real robot to perform mobile manipulation tasks.
• Figure 3: Examples of the multimodal LLM-based planner. (a) presents a no-failure case for the given command. (b) and (c) depict the failure reasoning ability of the model considering the inputs, for vision and feasibility failures respectively. (d) showcases the ability to recover from the failure in (b) with human instruction. In the replanning case (d), < history> tracks the initial request and the latest failure found.
• Figure 4: Success rates of failure explanation.
• Figure 5: Success rates of failure recovery.