Riemann-based Multi-scale Attention Reasoning Network for Text-3D Retrieval

TL;DR

This work tackles text-3D retrieval, a challenging cross-modal task hampered by scarce paired data and irregular 3D geometries. It introduces RMARN, a Riemannian-based multi-scale attention framework that refines text and point-cloud features with Adaptive Feature Refiners and computes cross-modal similarity via a combination of Riemann Local Similarity and Global Pooling Similarity, supplemented by Similarity Convolution and a Low Rank Filter. A new large-scale dataset, T3DR-HIT, provides 3,380 text-point cloud pairs spanning indoor scenes and Chinese artifacts to enable robust evaluation. Experiments show RMARN achieves superior retrieval performance and robust ablations highlight the importance of GPS, AFR, RLS, and the learned manifold parameters, underscoring the value of manifold-aware cross-modal alignment for 3D-text understanding. The work advances cross-modal retrieval by integrating Riemannian geometry with multi-scale similarity, offering practical benefits for AR/VR and large-scale 3D data management.

Abstract

Due to the challenges in acquiring paired Text-3D data and the inherent irregularity of 3D data structures, combined representation learning of 3D point clouds and text remains unexplored. In this paper, we propose a novel Riemann-based Multi-scale Attention Reasoning Network (RMARN) for text-3D retrieval. Specifically, the extracted text and point cloud features are refined by their respective Adaptive Feature Refiner (AFR). Furthermore, we introduce the innovative Riemann Local Similarity (RLS) module and the Global Pooling Similarity (GPS) module. However, as 3D point cloud data and text data often possess complex geometric structures in high-dimensional space, the proposed RLS employs a novel Riemann Attention Mechanism to reflect the intrinsic geometric relationships of the data. Without explicitly defining the manifold, RMARN learns the manifold parameters to better represent the distances between text-point cloud samples. To address the challenges of lacking paired text-3D data, we have created the large-scale Text-3D Retrieval dataset T3DR-HIT, which comprises over 3,380 pairs of text and point cloud data. T3DR-HIT contains coarse-grained indoor 3D scenes and fine-grained Chinese artifact scenes, consisting of 1,380 and over 2,000 text-3D pairs, respectively. Experiments on our custom datasets demonstrate the superior performance of the proposed method. Our code and proposed datasets are available at \url{https://github.com/liwrui/RMARN}.

Figures (4)

• Figure 1: Directly calculating the cosine similarity between two vectors may result in vectors with different meanings at different positions having the same similarity. However, in Riemannian geometry, vector movement conforms to the properties of the manifold, which can mitigate this problem.
• Figure 2: The overall architecture of RMARN proposed in this article. The Global Pooling Similarity module directly calculates the cosine distance between the text feature sequence and the point cloud feature sequence after pooling, while the Riemann Local Similarity module considers the point cloud sequence and the text sequence as two fields on a manifold and calculates the similarity between any two token pairs. Among them, $T_\mu^{(P_i)}$ and $P_\mu^{(Q_i)}$ represent the i-th token of the text and point cloud feature sequence, respectively.
• Figure 3: Examples of text point cloud pairs in The Elephant Meta Dataset. Each point cloud describes a fine-grained 3D object, and each point cloud corresponds to a caption consisting of 3 or more sentences that describe the specific content of the point cloud in natural language.
• Figure 4: Impact of different Nhead and low rank settings.