Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Voices to Worlds: Developing an AI-Powered Framework for 3D Object Generation in Augmented Reality

Majid Behravan, Denis Gracanin

TL;DR

This work presents Matrix, an open source AI powered framework for real-time speech driven 3D object generation in AR. It integrates multilingual speech to text, a text to 3D generator, and LLM based semantic understanding to deliver context aware object creation and recommendations with reduced latency. Key contributions include a memory efficient mesh simplification pipeline, a pre generated object repository with vector based retrieval, and a modular architecture suitable for local deployment on devices like HoloLens 2. The results demonstrate feasible real time AR generation with substantial reductions in mesh size and improved responsiveness, with potential applications in education, design, and accessibility, and it outlines future directions including image to 3D conversion and multimodal AR reasoning.

Abstract

This paper presents Matrix, an advanced AI-powered framework designed for real-time 3D object generation in Augmented Reality (AR) environments. By integrating a cutting-edge text-to-3D generative AI model, multilingual speech-to-text translation, and large language models (LLMs), the system enables seamless user interactions through spoken commands. The framework processes speech inputs, generates 3D objects, and provides object recommendations based on contextual understanding, enhancing AR experiences. A key feature of this framework is its ability to optimize 3D models by reducing mesh complexity, resulting in significantly smaller file sizes and faster processing on resource-constrained AR devices. Our approach addresses the challenges of high GPU usage, large model output sizes, and real-time system responsiveness, ensuring a smoother user experience. Moreover, the system is equipped with a pre-generated object repository, further reducing GPU load and improving efficiency. We demonstrate the practical applications of this framework in various fields such as education, design, and accessibility, and discuss future enhancements including image-to-3D conversion, environmental object detection, and multimodal support. The open-source nature of the framework promotes ongoing innovation and its utility across diverse industries.

From Voices to Worlds: Developing an AI-Powered Framework for 3D Object Generation in Augmented Reality

TL;DR

This work presents Matrix, an open source AI powered framework for real-time speech driven 3D object generation in AR. It integrates multilingual speech to text, a text to 3D generator, and LLM based semantic understanding to deliver context aware object creation and recommendations with reduced latency. Key contributions include a memory efficient mesh simplification pipeline, a pre generated object repository with vector based retrieval, and a modular architecture suitable for local deployment on devices like HoloLens 2. The results demonstrate feasible real time AR generation with substantial reductions in mesh size and improved responsiveness, with potential applications in education, design, and accessibility, and it outlines future directions including image to 3D conversion and multimodal AR reasoning.

Abstract

This paper presents Matrix, an advanced AI-powered framework designed for real-time 3D object generation in Augmented Reality (AR) environments. By integrating a cutting-edge text-to-3D generative AI model, multilingual speech-to-text translation, and large language models (LLMs), the system enables seamless user interactions through spoken commands. The framework processes speech inputs, generates 3D objects, and provides object recommendations based on contextual understanding, enhancing AR experiences. A key feature of this framework is its ability to optimize 3D models by reducing mesh complexity, resulting in significantly smaller file sizes and faster processing on resource-constrained AR devices. Our approach addresses the challenges of high GPU usage, large model output sizes, and real-time system responsiveness, ensuring a smoother user experience. Moreover, the system is equipped with a pre-generated object repository, further reducing GPU load and improving efficiency. We demonstrate the practical applications of this framework in various fields such as education, design, and accessibility, and discuss future enhancements including image-to-3D conversion, environmental object detection, and multimodal support. The open-source nature of the framework promotes ongoing innovation and its utility across diverse industries.

Paper Structure

This paper contains 21 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Comparison with Similar Works
  4. Framework Overview and Implementation
  5. Language Selection
  6. Command Initiation
  7. Translation and Transcription
  8. Object Extraction
  9. Object Recommendation
  10. Search Repository
  11. Object Generation and Mesh Simplification
  12. Interactive Environment Customization
  13. User Study and Performance Evaluation
  14. Tasks
  15. Results
  16. ...and 6 more sections

Figures (3)

  • Figure 1: The overview of the developed framework and the individual services provided.
  • Figure 2: A plate containing a banana, a red apple, a pear, and grapes, generated in real time using Matrix, is placed on a table by the user. The objects were created based on five voice commands given by the user: “Matrix, create a banana,” “Matrix, create a red apple,” “Matrix, create a pear,” “Matrix, create grapes,” and “Matrix, create a white plate.” The plate was retrieved from the repository, while the other objects were generated using text-to-3D conversion.
  • Figure 3: Status board example in Matrix AR application. The "what's happening" board displays the "Matrix offers" message, informing users to choose from requested objects, LLM recommendations, or items from the repository to create a 3D model.