3DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scale

TL;DR

The paper tackles the challenge of fine-grained 3D-text alignment at scale by introducing 3DAlign-DAER, a unified framework that combines a Dynamic Attention Policy (DAP) with a Hierarchical Attention Fusion (HAF) module and an Efficient Retrieval Strategy (ERS). A MCTS-guided refinement process optimizes token-to-point attentions during training, yielding highly discriminative cross-modal embeddings, while ERS enables scalable large-scale retrieval. To support this, Align3D-2M, a 2M-pair text-3D dataset, is constructed with automated generation and rigorous filtering. Extensive experiments demonstrate state-of-the-art performance across zero-shot classification, cross-modal retrieval, and open-world recognition, with notable gains in large-scale and few-shot settings, and ablations validate the contributions of MCTS optimization, the hybrid reward, and ERS. The work provides both a strong methodological advance and a valuable dataset/resource for the community.

Abstract

Despite recent advancements in 3D-text cross-modal alignment, existing state-of-the-art methods still struggle to align fine-grained textual semantics with detailed geometric structures, and their alignment performance degrades significantly when scaling to large-scale 3D databases. To overcome this limitation, we introduce 3DAlign-DAER, a unified framework designed to align text and 3D geometry via the proposed dynamic attention policy and the efficient retrieval strategy, capturing subtle correspondences for diverse cross-modal retrieval and classification tasks. Specifically, during the training, our proposed dynamic attention policy (DAP) employs the Hierarchical Attention Fusion (HAF) module to represent the alignment as learnable fine-grained token-to-point attentions. To optimize these attentions across different tasks and geometric hierarchies, our DAP further exploits the Monte Carlo tree search to dynamically calibrate HAF attention weights via a hybrid reward signal and further enhances the alignment between textual descriptions and local 3D geometry. During the inference, our 3DAlign-DAER introduces an Efficient Retrieval Strategy (ERS) to leverage efficient hierarchical searching in the large-scale embedding spaces, outperforming traditional methods (e.g., KNN) in accuracy and efficiency. Furthermore, to facilitate text-3D alignment research and train our 3DAlign-DAER, we construct Align3D-2M, a large-scale dataset featuring 2M text-3D pairs, to provide sufficient fine-grained cross-modal annotations. Extensive and comprehensive experiments demonstrate the superior performance of our 3DAlign-DAER on diverse benchmarks. We will release our codes, models, and datasets.

Figures (6)

• Figure 1: Our 3DAlign-DAER outperforms all task-specialized state-of-the-art methods on multiple 3D benchmarks and tasks (i.e., few/zero-shot classification, large-scale retrieval, and open-world understanding).
• Figure 2: Overview of the 3DAlign-DAER framework. (A) Training Phase: Input modalities are initially processed by pre-trained encoders from Uni3D-g. Features are then refined through the Hierarchical Attention Fusion (HAF) module and an attention optimization module (based on MCTS) that implements the Dynamic Attention Policy (DAP). Our 3DAlign-DAER is trained using a contrastive loss on the final aggregated embeddings. (B) Inference Phase: For retrieval tasks, queries are encoded using the point cloud and text encoders belonging to the trained 3DAlign-DAER model. We propose an Efficient Retrieval Strategy (ERS) to navigate semantic and spatial categories to find the target.
• Figure 3: Names and Sizes of Different Source Datasets Comprising Align3D-2M. Please note that we only selected relevant subsets or high-quality portions from each source.
• Figure 4: Attention Heatmap Visualization Comparison.
• Figure 5: Few-shot linear probing performance comparison on Objaverse-LVIS. Top-1 accuracy is reported for varying numbers of training samples per class (shots). Our 3DAlign-DAER outperforms all baselines across all shot settings.