Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling

TL;DR

The paper introduces Articulate AnyMesh, an open-vocabulary pipeline that converts arbitrary 3D meshes into articulated counterparts. It combines Movable Part Segmentation with PartSlip++ and open-vocabulary cues, geometry-aware visual prompting with GPT-4o for joint parameter estimation, and optional post-processing with Holopart and Meshy for shape completion and texturing. The approach demonstrates broad generalization beyond standard datasets, enabling large-scale articulated-object generation, annotation of existing meshes, and sim-to-real policy learning improvements. This work expands open-vocabulary capabilities in articulated-object modeling and provides a practical path toward richer simulation data and real-robot manipulation transfer.

Abstract

3D articulated objects modeling has long been a challenging problem, since it requires to capture both accurate surface geometries and semantically meaningful and spatially precise structures, parts, and joints. Existing methods heavily depend on training data from a limited set of handcrafted articulated object categories (e.g., cabinets and drawers), which restricts their ability to model a wide range of articulated objects in an open-vocabulary context. To address these limitations, we propose Articulate Anymesh, an automated framework that is able to convert any rigid 3D mesh into its articulated counterpart in an open-vocabulary manner. Given a 3D mesh, our framework utilizes advanced Vision-Language Models and visual prompting techniques to extract semantic information, allowing for both the segmentation of object parts and the construction of functional joints. Our experiments show that Articulate Anymesh can generate large-scale, high-quality 3D articulated objects, including tools, toys, mechanical devices, and vehicles, significantly expanding the coverage of existing 3D articulated object datasets. Additionally, we show that these generated assets can facilitate the acquisition of new articulated object manipulation skills in simulation, which can then be transferred to a real robotic system. Our Github website is https://articulate-anymesh.github.io.

Figures (8)

• Figure 1: Articulate AnyMesh turns 3D meshes (e.g. retrieved from Objaverse deitke2024objaverse) into articulated objects. Our pipeline is capable of processing a wide range of objects, including daily necessities, furniture, vehicles and even fictional objects.
• Figure 2: Articulate AnyMesh converts 3D meshes from various sources to high quality articulated objects via three main parts: A. Movable Part Segmentation, B. Articulation Estimation, C. Post-Processing.
• Figure 3: Results of articulated object generation. Each example shows (from left to right): the raw surface mesh generated by Meshy, the completed geometry, and the textured object, all rendered under two different poses.
• Figure 4: Results of Articulate AnyMesh applied to hand-crafted meshes retrieved from Objaverse.
• Figure 5: We build the digital twins of real-world objects in the simulation using our method. Then we sample trajectories completing the tasks in the simulation and replicate those trajectories in the real-world.