Part-X-MLLM: Part-aware 3D Multimodal Large Language Model

TL;DR

Part-X-MLLM introduces a native 3D part-aware multimodal LLM that outputs an executable, part-based program encoding bounding boxes, persistent references, semantics, and edit commands from RGB point clouds and prompts. A dual-encoder architecture decouples structure and appearance, and a grammar-driven autoregressive decoder emits a unified plan that can drive arbitrary geometry engines for generation, editing, and reasoning. The authors establish UniPart-Bench, a 30k-entry, part-centric benchmark, and demonstrate state-of-the-art grounding, compositional generation, and localized editing across 11 task families with strong IoU, recall, and language-based metrics. By treating parts as first-class citizens and providing a language-native, model-agnostic interface, Part-X-MLLM enables precise, auditable, and semantically controllable 3D asset manipulation across diverse categories.

Abstract

We introduce Part-X-MLLM, a native 3D multimodal large language model that unifies diverse 3D tasks by formulating them as programs in a structured, executable grammar. Given an RGB point cloud and a natural language prompt, our model autoregressively generates a single, coherent token sequence encoding part-level bounding boxes, semantic descriptions, and edit commands. This structured output serves as a versatile interface to drive downstream geometry-aware modules for part-based generation and editing. By decoupling the symbolic planning from the geometric synthesis, our approach allows any compatible geometry engine to be controlled through a single, language-native frontend. We pre-train a dual-encoder architecture to disentangle structure from semantics and instruction-tune the model on a large-scale, part-centric dataset. Experiments demonstrate that our model excels at producing high-quality, structured plans, enabling state-of-the-art performance in grounded Q\&A, compositional generation, and localized editing through one unified interface. Project page: https://chunshi.wang/Part-X-MLLM/

Figures (11)

• Figure 1: Part-X-MLLM is a natively 3D, part-aware multimodal large language model that provides comprehensive understanding of 3D shapes and supports a wide range of 3D understanding tasks. It also seamlessly integrates with diffusion-based pipelines, enabling semantically precise part-aware 3D shape generation and editing.
• Figure 2: The Part-X-MLLM Framework. Our pipeline begins by encoding geometry and appearance features separately using a dual-encoder architecture, which are then fused together with text prompts. These combined features are passed to an autoregressive decoder that generates a program-like token sequence representing a plan (e.g., bounding boxes, edit commands). Finally, specialized geometry heads execute this plan to enable part-aware generation and editing.
• Figure 3: Task realization with a planning language. A decoder outputs program tokens that unify diverse interactions: (Top) part-aware generation guided by bounding boxes; (Middle) grounded Q&A whose answers embed BBox tokens; (Bottom) auto-located 3D editing executed via cuboid masks and commands. The numbered circles (e.g., ) denote the corresponding task types.
• Figure 4: Qualitative shape decomposition results.
• Figure 5: Qualitative results for part-aware editing. Our model successfully interprets natural language instructions to perform localized edits, while preserving the integrity of the original object.