SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentation

TL;DR

This work tackles data scarcity in cross-modal 3D retrieval by introducing SCA3D, an online data augmentation framework that uses a multimodal language model to caption segmented 3D shape parts and assemble diverse part-based shapes. It pairs generated 3D shapes with captions through inter- and intra-part distance adjustments, and learns robust embeddings with PointNet-based shape encodings and a bidirectional GRU text encoder, using Earth Mover's Distance to perform fine-grained cross-modal matching plus InfoNCE-based contrastive learning for bidirectional retrieval. Empirical results on Text2Shape show substantial gains over prior methods in both Shape-to-Text and Text-to-Shape tasks, validating the effectiveness of part-level data augmentation and EMD-based alignment. The approach promises improved data efficiency and generalization for robot perception and embodied AI, with potential extensions to more complex 3D scenes and dynamic environments.

Abstract

The cross-modal 3D retrieval task aims to achieve mutual matching between text descriptions and 3D shapes. This has the potential to enhance the interaction between natural language and the 3D environment, especially within the realms of robotics and embodied artificial intelligence (AI) applications. However, the scarcity and expensiveness of 3D data constrain the performance of existing cross-modal 3D retrieval methods. These methods heavily rely on features derived from the limited number of 3D shapes, resulting in poor generalization ability across diverse scenarios. To address this challenge, we introduce SCA3D, a novel 3D shape and caption online data augmentation method for cross-modal 3D retrieval. Our approach uses the LLaVA model to create a component library, captioning each segmented part of every 3D shape within the dataset. Notably, it facilitates the generation of extensive new 3D-text pairs containing new semantic features. We employ both inter and intra distances to align various components into a new 3D shape, ensuring that the components do not overlap and are closely fitted. Further, text templates are utilized to process the captions of each component and generate new text descriptions. Besides, we use unimodal encoders to extract embeddings for 3D shapes and texts based on the enriched dataset. We then calculate fine-grained cross-modal similarity using Earth Mover's Distance (EMD) and enhance cross-modal matching with contrastive learning, enabling bidirectional retrieval between texts and 3D shapes. Extensive experiments show our SCA3D outperforms previous works on the Text2Shape dataset, raising the Shape-to-Text RR@1 score from 20.03 to 27.22 and the Text-to-Shape RR@1 score from 13.12 to 16.67. Codes can be found in https://github.com/3DAgentWorld/SCA3D.

Figures (5)

• Figure 1: The overview of our proposed SCA3D. It consists of three components: the 3D-caption paired data augmentation module, unimodal encoders, and the matching module. The 3D-caption paired data augmentation module continuously creates extensive 3D-text pairs with diverse geometry and semantics to facilitate cross-modal training. The unimodal encoders comprise a 3D shape encoder and a text encoder, which learn 3D shape and text embeddings from the input data. The matching module computes similarity scores between each 3D-text pair using Earth Mover's Distance (EMD), maximizing the similarity of positive pairs while minimizing the similarity of negative pairs.
• Figure 2: The pipeline of 3D-caption paired data augmentation. The component library is created by captioning 3D shape parts through LLaVA. During training, different components are sampled from this library, and repositioning is applied to generate new 3D shapes with correct geometry and corresponding text captions.
• Figure 3: Generated 3D shapes and captions through data augmentation.
• Figure 4: Shape-to-text retrieval results. Each query shape is displayed with the top-5 ranked texts. Ground truths are highlighted in red.
• Figure 5: Text-to-shape retrieval results. Each query text is displayed with the top-5 ranked shapes. Ground truths are indicated as GT.