Transcending Dimensions using Generative AI: Real-Time 3D Model Generation in Augmented Reality

TL;DR

This work addresses the challenge of making real-time 3D content creation in augmented reality accessible to non-experts by integrating generative AI with AR in a unified pipeline. It combines 2D-to-3D conversion using Shap-E with robust object detection via Mask R-CNN, a zone-based capture workflow, and per-object cropping to isolate items before generation, enabling AR-ready 3D models to be rendered in real time. The study reports a System Usability Scale of 69.64 overall, with higher satisfaction (80.71) among participants familiar with AR/VR, and provides detailed performance metrics showing real-time processing viability on typical AR hardware. The approach demonstrates potential across gaming, education, and retail by democratizing 3D model creation and supporting interactive, on-demand object visualization in AR environments.

Abstract

Traditional 3D modeling requires technical expertise, specialized software, and time-intensive processes, making it inaccessible for many users. Our research aims to lower these barriers by combining generative AI and augmented reality (AR) into a cohesive system that allows users to easily generate, manipulate, and interact with 3D models in real time, directly within AR environments. Utilizing cutting-edge AI models like Shap-E, we address the complex challenges of transforming 2D images into 3D representations in AR environments. Key challenges such as object isolation, handling intricate backgrounds, and achieving seamless user interaction are tackled through advanced object detection methods, such as Mask R-CNN. Evaluation results from 35 participants reveal an overall System Usability Scale (SUS) score of 69.64, with participants who engaged with AR/VR technologies more frequently rating the system significantly higher, at 80.71. This research is particularly relevant for applications in gaming, education, and AR-based e-commerce, offering intuitive, model creation for users without specialized skills.

Figures (13)

• Figure 1: From a 2D image to an AR environment. Left half: From a 2D image on a computer screen to a 3D object in an AR environment. Right half: From a 2D image of a real world object to a 3D object in an AR environment.
• Figure 2: Challenges in AR headset views and Shap-E model outputs.
• Figure 3: Cloning object workflow.
• Figure 4: AR hand menu.
• Figure 5: Drawing selection zone (Lasso selection interaction technique).