MagicCraft: Natural Language-Driven Generation of Dynamic and Interactive 3D Objects for Commercial Metaverse Platforms

TL;DR

MagicCraft addresses the barrier to creating dynamic interactive 3D objects in metaverse platforms by orchestrating an end-to-end AI-based pipeline from natural language prompts to platform-ready assets. It combines image generation, image-to-3D reconstruction with real-world scaling, automatic behavior scripting, and an interactive refinement UI, deployed on Cluster. A user study with 7 expert designers and 51 general users shows substantial time savings and lowered skill requirements, though texture and motion quality require further improvement. The work demonstrates the potential of AI-assisted 3D content creation to broaden participation and speed up prototyping in commercial metaverse ecosystems.

Abstract

Metaverse platforms are rapidly evolving to provide immersive spaces for user interaction and content creation. However, the generation of dynamic and interactive 3D objects remains challenging due to the need for advanced 3D modeling and programming skills. To address this challenge, we present MagicCraft, a system that generates functional 3D objects from natural language prompts for metaverse platforms. MagicCraft uses generative AI models to manage the entire content creation pipeline: converting user text descriptions into images, transforming images into 3D models, predicting object behavior, and assigning necessary attributes and scripts. It also provides an interactive interface for users to refine generated objects by adjusting features such as orientation, scale, seating positions, and grip points. Implemented on Cluster, a commercial metaverse platform, MagicCraft was evaluated by 7 expert CG designers and 51 general users. Results show that MagicCraft significantly reduces the time and skill required to create 3D objects. Users with no prior experience in 3D modeling or programming successfully created complex, interactive objects and deployed them in the metaverse. Expert feedback highlighted the system's potential to improve content creation workflows and support rapid prototyping. By integrating AI-generated content into metaverse platforms, MagicCraft makes 3D content creation more accessible.

Figures (11)

• Figure 1: When the user describes the object to be created in natural language, an image of the object is generated. A 3D object is then created from the generated image. The user sets the behavior, such as the position where the object will sit. The object with the set behavior is immediately uploaded to the metaverse space and can be used by multiple people.
• Figure 2: The initial user input is augmented by a large language model (LLM), and this augmented input is used as input for an AI to generate an image. The resulting image is used as input for an AI to generate a 3D model, and also helps to estimate the real-world size of the object. The 3D model is scaled based on the estimated size, and users can adjust the object's behavior within the system. The finished object can be immediately uploaded to metaverse services, where users can place or interact with it in virtual space. Users can also download the object as a local file.
• Figure 3: Position Adjustment User Interfaces
• Figure 4: MagicCraft system architecture illustrating the end-to-end workflow across four sequential phases: (a) image generation phase, (b) 3D generation phase, (c) optional script generation phase for behavioral programming, and (d) platform upload phase for cluster integration. Data flows are shown between user interactions (left), web application processing (center), and specialized AI services and platform integration (right).
• Figure 5: The interface of MagicCraft web application. (A) Initial prompt input screen with API token entry field and text description input for object generation. (B) Generated image preview with options to accept or regenerate. (C) 3D model adjustment interface with controllable parameters for position, rotation, scale, and interactive points (sitting/grabbing). The mannequin provides visual reference for avatar interaction. (D) Upload interface with naming fields and confirmation options for deploying the 3D object to the Cluster metaverse platform.