3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V

TL;DR

This paper tackles the limited 3D spatial understanding of GPT-4V by introducing 3DAxiesPrompts (3DAP), a visual prompting method that overlays a 3D coordinate system with scale information onto input images. The authors define a structured 3D geometry framework and demonstrate its integration into prompts to markedly improve 3D reasoning across three tasks: 2D to 3D point reconstruction, 2D to 3D point matching, and 3D object detection, validated on the newly created 3DAP-Data dataset. Key contributions include the 3DAP prompting method, a dedicated 3D visual prompting dataset, and ablation studies confirming the value of explicit coordinate axes and scale markers. The work advances practical 3D perception for multimodal models, with potential impact on domains such as autonomous systems, robotics, and AR/VR where accurate 3D reasoning is essential.

Abstract

In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.

Figures (12)

• Figure 1: Using the 3DAP method proposed in this paper, GPT-4v accurately answers the height of the stool, the direction axis, and the length of the stool leg.
• Figure 2: 3DAP Example diagram of specific steps in object labeling
• Figure 3: Diagram of left and right-hand coordinate system
• Figure 4: Comparisons of GPT-4V prompting in the task of 2D to 3D Point Reconstruction: (left)we input images that exclusively highlight key point information. Aims to discern the relative positional coordinates of the remaining points. GPT-4V suggests an enhancement: encouraging users to include more comprehensive annotation details, such as directions in the three-dimensional coordinate system and the proportionality of the coordinates, (right) We input 3D images marked with 3DAP, With the coordinate information of a specific point, GPT-4V demonstrates remarkable precision in inferring the relative direction and position coordinates of other points.
• Figure 5: Comparisons of GPT-4V prompting in the task of matching points from 2D to 3D: (left) We input three images with only marked points, GPT-4V's understanding of spatial information is not accurate, (right) we input one 3D image labeled with 3DAP method and two 2D images, mark the point, GPT-4V employs its analytical capabilities to ascertain the position of this key point within the 3D coordinate framework, and finds the key point corresponding to the position relationship in the two 2D images.