Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AIR-Embodied: An Efficient Active 3DGS-based Interaction and Reconstruction Framework with Embodied Large Language Model

Zhenghao Qi, Shenghai Yuan, Fen Liu, Haozhi Cao, Tianchen Deng, Jianfei Yang, Lihua Xie

TL;DR

AIR-Embodied is a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction, providing a robust solution to challenges in active 3D reconstruction.

Abstract

Recent advancements in 3D reconstruction and neural rendering have enhanced the creation of high-quality digital assets, yet existing methods struggle to generalize across varying object shapes, textures, and occlusions. While Next Best View (NBV) planning and Learning-based approaches offer solutions, they are often limited by predefined criteria and fail to manage occlusions with human-like common sense. To address these problems, we present AIR-Embodied, a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction. AIR-Embodied utilizes a three-stage process: understanding the current reconstruction state via multi-modal prompts, planning tasks with viewpoint selection and interactive actions, and employing closed-loop reasoning to ensure accurate execution. The agent dynamically refines its actions based on discrepancies between the planned and actual outcomes. Experimental evaluations across virtual and real-world environments demonstrate that AIR-Embodied significantly enhances reconstruction efficiency and quality, providing a robust solution to challenges in active 3D reconstruction.

AIR-Embodied: An Efficient Active 3DGS-based Interaction and Reconstruction Framework with Embodied Large Language Model

TL;DR

AIR-Embodied is a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction, providing a robust solution to challenges in active 3D reconstruction.

Abstract

Recent advancements in 3D reconstruction and neural rendering have enhanced the creation of high-quality digital assets, yet existing methods struggle to generalize across varying object shapes, textures, and occlusions. While Next Best View (NBV) planning and Learning-based approaches offer solutions, they are often limited by predefined criteria and fail to manage occlusions with human-like common sense. To address these problems, we present AIR-Embodied, a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction. AIR-Embodied utilizes a three-stage process: understanding the current reconstruction state via multi-modal prompts, planning tasks with viewpoint selection and interactive actions, and employing closed-loop reasoning to ensure accurate execution. The agent dynamically refines its actions based on discrepancies between the planned and actual outcomes. Experimental evaluations across virtual and real-world environments demonstrate that AIR-Embodied significantly enhances reconstruction efficiency and quality, providing a robust solution to challenges in active 3D reconstruction.
Paper Structure (23 sections, 11 equations, 5 figures, 3 tables)

This paper contains 23 sections, 11 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Active Reconstruction with Radiance Fields
  4. Vision Tasks Enhanced by Embodiment
  5. LLM for robotic tasks
  6. Proposed Method
  7. Problem Definition
  8. Viewpoint and Manipulation Planning
  9. Reconstruction Model Representation
  10. Gaussian Splatting
  11. Pixcel-level Uncertainty
  12. AIR-Embodied Framework
  13. High-level Reasoning
  14. Novel Viewpoint Synthesis
  15. Manipulation Synthesis
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Overview. Previous NBV methods rely on low-level uncertainty and limited viewpoint selection. Our system uses embodied agents for high-level understanding, enabling free-space viewpoint planning and interactive manipulation. Closed-loop reasoning corrects action errors, achieving generalized, high-quality object reconstructions.
  • Figure 2: Overview of AIR-Embodied. In stage I, the agent derives high-level understanding from multi-modal low-level data and maps it to 3D space. In stage II, additional reasoning and constraints are added while generating plans for new viewpoints and interactive actions. In stage III, actions are executed, and closed-loop reasoning corrects any errors.
  • Figure 3: Close Loop Reasoning. Compare the operational results with the desired target state and propose appropriate fine-tuning policy.
  • Figure 4: AIR-Embodied: active reasoning while scanning.
  • Figure 5: Qualitative Comparison. The proposed method scans better than the current SOTA.