VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding

TL;DR

The paper addresses the problem of converting 2D visual inputs into accurate 3D representations within a flexible, multimodal framework. It introduces VisionGPT-3D, which integrates SOTA vision models (e.g., SAM, YOLO, DINO) and monocular depth estimation (MiDaS) to generate depth maps, derive point clouds, reconstruct meshes, and optionally synthesize videos, guided by AI-driven algorithm selection at each stage. It contributes an end-to-end, task-aware pipeline for depth-based 3D reconstruction and validation, blending traditional computer vision methods with multimodal foundation models. The work aims to enable automated, adaptable 3D scene understanding and video generation from multimodal inputs, with potential applications in AR/VR, robotics, and content creation.

Abstract

The evolution of text to visual components facilitates people's daily lives, such as generating image, videos from text and identifying the desired elements within the images. Computer vision models involving the multimodal abilities in the previous days are focused on image detection, classification based on well-defined objects. Large language models (LLMs) introduces the transformation from nature language to visual objects, which present the visual layout for text contexts. OpenAI GPT-4 has emerged as the pinnacle in LLMs, while the computer vision (CV) domain boasts a plethora of state-of-the-art (SOTA) models and algorithms to convert 2D images to their 3D representations. However, the mismatching between the algorithms with the problem could lead to undesired results. In response to this challenge, we propose an unified VisionGPT-3D framework to consolidate the state-of-the-art vision models, thereby facilitating the development of vision-oriented AI. VisionGPT-3D provides a versatile multimodal framework building upon the strengths of multimodal foundation models. It seamlessly integrates various SOTA vision models and brings the automation in the selection of SOTA vision models, identifies the suitable 3D mesh creation algorithms corresponding to 2D depth maps analysis, generates optimal results based on diverse multimodal inputs such as text prompts. Keywords: VisionGPT-3D, 3D vision understanding, Multimodal agent

Figures (8)

• Figure 1: Image depth map
• Figure 2: Depth of 2D images from different angles
• Figure 3: 2D image key points
• Figure 4: AI based 2D-3D image converter pipeline
• Figure 5: Incorrect generated mesh