3D-Agent:Tri-Modal Multi-Agent Collaboration for Scalable 3D Object Annotation

TL;DR

Tri-MARF presents a tri-modal, tri-agent framework for scalable 3D object annotation by integrating 2D multi-view images, textual descriptions, and 3D point clouds through a four-stage pipeline (data preparation, VLM annotation, information aggregation with MAB-based selection, and gating). The approach achieves state-of-the-art cross-modal alignment (high CLIPScore and ViLT R@5) and high throughput (up to 12k objects/hour on a single NVIDIA A100), with robust generalization across diverse 3D datasets and occlusion conditions. Key contributions include a reinforcement-learning-based aggregation that balances textual confidence and visual grounding, a cosine-similarity gating mechanism to align text with geometry, and a systematic ablation program that isolates the benefits of multi-view, multi-agent collaboration. The results demonstrate Tri-MARF’s effectiveness for large-scale 3D annotation tasks relevant to autonomous driving, robotics, and AR, and the work offers practical insights into parameter choices, memory usage, and potential scalability improvements.

Abstract

Driven by applications in autonomous driving robotics and augmented reality 3D object annotation presents challenges beyond 2D annotation including spatial complexity occlusion and viewpoint inconsistency Existing approaches based on single models often struggle to address these issues effectively We propose Tri MARF a novel framework that integrates tri modal inputs including 2D multi view images textual descriptions and 3D point clouds within a multi agent collaborative architecture to enhance large scale 3D annotation Tri MARF consists of three specialized agents a vision language model agent for generating multi view descriptions an information aggregation agent for selecting optimal descriptions and a gating agent that aligns textual semantics with 3D geometry for refined captioning Extensive experiments on Objaverse LVIS Objaverse XL and ABO demonstrate that Tri MARF substantially outperforms existing methods achieving a CLIPScore of 88 point 7 compared to prior state of the art methods retrieval accuracy of 45 point 2 and 43 point 8 on ViLT R at 5 and a throughput of up to 12000 objects per hour on a single NVIDIA A100 GPU

Figures (21)

• Figure 1: Comparison example of our Tri-MARF captions with previous SOTA methods. Our Tri-MARF not only accurately recognizes the specific names of objects, but also provides rich and correct details. Some keywords in the annotations are shown in red, and the specific names of the objects are shown in orange. Please note that only our Tri-MARF can mark them out.
• Figure 2: The illustration of our Tri-MARF for 3D object annotation, featuring a collaborative multi-agent mechanism. The process starts with Agent 1 (VLM Annotation Agent), which uses a visual language model (e.g., Qwen2.5-VL-72B-Instruct) to generate 5 text descriptions for each view of a 3D object from six standard viewpoints (front, back, left, right, top, bottom). These descriptions are then processed by Agent 2 (Information Aggregation Agent), which uses RoBERTa+DBSCAN for semantic embedding clustering, CLIP for visual-text alignment, and integrates a multi-armed bandit (MAB) model to optimize description selection and balance exploration and exploitation to obtain the final captions. Agent 3 (Point Cloud Gating Agent) uses threshold control to align text and 3D point clouds, further reducing the wrong results produced by VLM annotations. Please note that our point cloud is a pre-rendered asset.
• Figure 3: Detailed demonstration of the gating agent of our Tri-MARF. The pre-trained Uni3d encoder is used to handle point cloud and text matching on the open domain.
• Figure 4: Classification accuracy of the four annotation methods on Objaverse-LVIS by using string matching and GPT-4o scoring.
• Figure 5: Comparison of CLIPScore trends with varying view number on Objaverse-LVIS (1k).