Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline

Wenhao Sun, Sai Hou, Zixuan Wang, Bo Yu, Shaoshan Liu, Xu Yang, Shuai Liang, Yiming Gan, Yinhe Han

TL;DR

DaDu-E rethinks robotic planning by pairing a lightweight LLM with a closed-loop pipeline that includes structured instruction sets, planning feedback, and memory augmentation. By constraining robot scope to fixed domains and enriching inputs with frequent visual feedback and memory, it achieves comparable task success rates to large-model planners like COME-Robot* while reducing computation by $6.6\times$, enabling local-server deployment. The approach demonstrates strong performance across four embodied AI task levels, with notable gains from feedback and memory in both real-world and simulated environments. This work highlights the practicality of resource-efficient, closed-loop embodied intelligence for real-world robotics and scalable deployment. Overall, DaDu-E offers a compelling pathway to high-performance robotic planning that balances efficiency, adaptability, and scalability.

Abstract

Performing complex tasks in open environments remains challenging for robots, even when using large language models (LLMs) as the core planner. Many LLM-based planners are inefficient due to their large number of parameters and prone to inaccuracies because they operate in open-loop systems. We think the reason is that only applying LLMs as planners is insufficient. In this work, we propose DaDu-E, a robust closed-loop planning framework for embodied AI robots. Specifically, DaDu-E is equipped with a relatively lightweight LLM, a set of encapsulated robot skill instructions, a robust feedback system, and memory augmentation. Together, these components enable DaDu-E to (i) actively perceive and adapt to dynamic environments, (ii) optimize computational costs while maintaining high performance, and (iii) recover from execution failures using its memory and feedback mechanisms. Extensive experiments on real-world and simulated tasks show that DaDu-E achieves task success rates comparable to embodied AI robots with larger models as planners like COME-Robot, while reducing computational requirements by $6.6 \times$. Users are encouraged to explore our system at: \url{https://rlc-lab.github.io/dadu-e/}.

DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline

TL;DR

DaDu-E rethinks robotic planning by pairing a lightweight LLM with a closed-loop pipeline that includes structured instruction sets, planning feedback, and memory augmentation. By constraining robot scope to fixed domains and enriching inputs with frequent visual feedback and memory, it achieves comparable task success rates to large-model planners like COME-Robot* while reducing computation by , enabling local-server deployment. The approach demonstrates strong performance across four embodied AI task levels, with notable gains from feedback and memory in both real-world and simulated environments. This work highlights the practicality of resource-efficient, closed-loop embodied intelligence for real-world robotics and scalable deployment. Overall, DaDu-E offers a compelling pathway to high-performance robotic planning that balances efficiency, adaptability, and scalability.

Abstract

Performing complex tasks in open environments remains challenging for robots, even when using large language models (LLMs) as the core planner. Many LLM-based planners are inefficient due to their large number of parameters and prone to inaccuracies because they operate in open-loop systems. We think the reason is that only applying LLMs as planners is insufficient. In this work, we propose DaDu-E, a robust closed-loop planning framework for embodied AI robots. Specifically, DaDu-E is equipped with a relatively lightweight LLM, a set of encapsulated robot skill instructions, a robust feedback system, and memory augmentation. Together, these components enable DaDu-E to (i) actively perceive and adapt to dynamic environments, (ii) optimize computational costs while maintaining high performance, and (iii) recover from execution failures using its memory and feedback mechanisms. Extensive experiments on real-world and simulated tasks show that DaDu-E achieves task success rates comparable to embodied AI robots with larger models as planners like COME-Robot, while reducing computational requirements by . Users are encouraged to explore our system at: \url{https://rlc-lab.github.io/dadu-e/}.

Paper Structure

This paper contains 36 sections, 6 equations, 4 figures, 16 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Long-horizon Task Planning via multi-modal LLMs.
  4. End-to-end Approach and Modular-based Solutions.
  5. Close-loop Control.
  6. Method
  7. Architecture
  8. Instruction Sets
  9. Planning Feedback
  10. Memory Augmentation
  11. Short-term memory
  12. Long-term memory
  13. Evaluation
  14. Evaluation Environment set-up
  15. Baseline methods
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Architecture of DaDu-E.
  • Figure 2: Overview of our real robot.
  • Figure 3: Overview of computing cost with baselines.
  • Figure 4: Overview of peak cost with end-to-end methods.