Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs

TL;DR

Idea23D tackles the challenge of generating textured 3D content from high-level, interleaved multimodal inputs (IDEAs) by orchestrating three LMM-based agents for prompt generation, draft selection, and feedback reflection, augmented by a memory module. The framework converts IDEA inputs into Text-to-Image prompts and then into 3D via an Image-to-3D pipeline, iterating with a memory-guided refinement loop to improve alignment. It introduces Eval3DAIGC-198, a dataset of 198 IDEAs used to quantify how well generated outputs match inputs, and demonstrates superior performance in success rate, accuracy, and user satisfaction across diverse LMM, T-2-I, and I-2-3D backbones, supported by a user study and ablation analyses. The results indicate that automated self-refinement accelerates convergence and yields outputs closer to ground truth than caption-based baselines, signaling strong practical impact for automated, multimodal 3D design tools.

Abstract

With the success of 2D diffusion models, 2D AIGC content has already transformed our lives. Recently, this success has been extended to 3D AIGC, with state-of-the-art methods generating textured 3D models from single images or text. However, we argue that current 3D AIGC methods still do not fully unleash human creativity. We often imagine 3D content made from multimodal inputs, such as what it would look like if my pet bunny were eating a doughnut on the table. In this paper, we explore a novel 3D AIGC approach: generating 3D content from IDEAs. An IDEA is a multimodal input composed of text, image, and 3D models. To our knowledge, this challenging and exciting 3D AIGC setting has not been studied before. We propose the new framework Idea23D, which combines three agents based on large multimodal models (LMMs) and existing algorithmic tools. These three LMM-based agents are tasked with prompt generation, model selection, and feedback reflection. They collaborate and critique each other in a fully automated loop, without human intervention. The framework then generates a text prompt to create 3D models that align closely with the input IDEAs. We demonstrate impressive 3D AIGC results that surpass previous methods. To comprehensively assess the 3D AIGC capabilities of Idea23D, we introduce the Eval3DAIGC-198 dataset, containing 198 multimodal inputs for 3D generation tasks. This dataset evaluates the alignment between generated 3D content and input IDEAs. Our user study and quantitative results show that Idea23D significantly improves the success rate and accuracy of 3D generation, with excellent compatibility across various LMM, Text-to-Image, and Image-to-3D models. Code and dataset are available at \url{https://idea23d.github.io/}.

Figures (11)

• Figure 1: The Idea23D framework synergizes the capabilities of the Large Multimodal Model (LMM), Text-to-Image (T-2-I), and Image-to-3D (I-2-3D) models to transform complex multimodal input IDEAs into tangible 3D models. This process begins with the user articulating high-level 3D design requirements (IDEA). Following this, the LMM generates textual prompts (Prompt Generation) that are then converted into 3D models. These models are evaluated through a Multiview Image Generation and Evaluation process, leading to the Selection of an Optimal 3D Model. Subsequently, the T-2-I prompt is refined (Feedback Generation) using insights from the LMM. Additionally, an integrated memory module (see Sec. \ref{['sec_Memory_Module']}), meticulously records each iteration, facilitating a multimodal, iterative self-refinement cycle within the framework. Note that this procedure is fully automatic without any human intervention.
• Figure 2: Overview of the framework of Idea23D, which employs LMM agents to unleash the T-2-3D model's potential through iterative self-refinement to provide better T-2-3D prompts for the input user IDEA. Green rounded rectangles indicate steps completed by LMM agents. Purple rounded rectangles indicate T-2-3D modules, including T-2-I models and I-2-3D models. The yellow rounded rectangle indicates the off-the-shelf 3D model multi-view generation algorithm. The blue color indicates the memory module, which saves all the feedback from previous rounds, the best 3D model, and the best text prompt. Note that this cycle is fully automatically executed by LMM agents, without any human intervention.
• Figure 3: Overview of 3D models generated from various types of multimodal IDEA inputs supported by Idea23D. The light red box on the left is the user input IDEA containing text, images and 3D models. In the center are the baseline results generated directly from the same T-2-I model with caption-based T-2-I prompt (see Sec. \ref{['Experimental_Setup']}). The model on the right is the result generated by iteratively self-refining the T-2-I prompts with Idea23D. Comparison with more existing methods is shown in Fig. \ref{['fig:c_models']}.
• Figure 4: Key module ablation across iterations. Note that the full model reaches satisfactory results (judged by the LMM agent 5 in Fig. \ref{['fig:overview_famework']}) at iteration 3. The illustrated experiment has no maximum iteration limit.
• Figure 5: IDEA Content Distribution.