BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretraining

TL;DR

BlendCLIP addresses the synthetic-to-real domain gap in zero-shot 3D object classification for outdoor driving by blending a large-scale synthetic CAD-based multimodal pretraining regime with real outdoor LiDAR–image–text triplets through a curriculum-based data-mixing strategy. Building on ULIP-2, it trains a 3D encoder to align with frozen CLIP image and text embeddings, while a dedicated outdoor triplet generation pipeline provides object-level realism. The approach achieves state-of-the-art zero-shot results on nuScenes and TruckScenes, preserves strong synthetic benchmarks like Objaverse-LVIS, and demonstrates notable label efficiency, with meaningful gains from as little as 1.5% real data per batch. This work highlights that principled domain adaptation, rather than extensive real-world annotation, is key to robust open-vocabulary 3D perception in autonomous driving.

Abstract

Zero-shot 3D object classification is crucial for real-world applications like autonomous driving, however it is often hindered by a significant domain gap between the synthetic data used for training and the sparse, noisy LiDAR scans encountered in the real-world. Current methods trained solely on synthetic data fail to generalize to outdoor scenes, while those trained only on real data lack the semantic diversity to recognize rare or unseen objects. We introduce BlendCLIP, a multimodal pretraining framework that bridges this synthetic-to-real gap by strategically combining the strengths of both domains. We first propose a pipeline to generate a large-scale dataset of object-level triplets -- consisting of a point cloud, image, and text description -- mined directly from real-world driving data and human annotated 3D boxes. Our core contribution is a curriculum-based data mixing strategy that first grounds the model in the semantically rich synthetic CAD data before progressively adapting it to the specific characteristics of real-world scans. Our experiments show that our approach is highly label-efficient: introducing as few as 1.5\% real-world samples per batch into training boosts zero-shot accuracy on the nuScenes benchmark by 27\%. Consequently, our final model achieves state-of-the-art performance on challenging outdoor datasets like nuScenes and TruckScenes, improving over the best prior method by 19.3\% on nuScenes, while maintaining strong generalization on diverse synthetic benchmarks. Our findings demonstrate that effective domain adaptation, not full-scale real-world annotation, is the key to unlocking robust open-vocabulary 3D perception. Our code and dataset will be released upon acceptance on https://github.com/kesu1/BlendCLIP.

Figures (8)

• Figure 1: Overview of BlendCLIP. We construct a large-scale set of outdoor LiDAR–image–text triplets from real-world driving data and introduce a curriculum-based data mixing strategy to integrate them with existing synthetic triplets. A 3D encoder is trained to align with frozen CLIP text and image embeddings, enabling robust zero-shot 3D classification across both synthetic and real-world domains.
• Figure 2: Overview of outdoor triplet generation. Given time-synchronized sensor data, we apply multi-sweep fusion and motion compensation to extract dense object-level point clouds from annotated 3D boxes. Image crops are obtained by projecting the box into synchronized camera views and selecting those with high visibility. Each image is then captioned using a vision-language model to form point cloud–image–text triplets used in training. Underlined denotes the subject, while bold indicates fine-grained descriptions.
• Figure 3: Comparison of top-1 accuracy on nuScenes-triplets validation set over the first 80 training epochs. Our method (BlendCLIP) exhibits a sharp performance jump at epoch 8, after adding just one sample per GPU, while the baseline trained only on synthetic data remains flat.
• Figure 4: 2D UMAP projection of CLIP-based 3D features on nuScenes-triplets validation set. (a) ULIP‑2 pretrained solely on synthetic CAD models shows poor class separation. (b) Our model, trained with data mixing, exhibits clearer clustering of real-world object categories. Colors denote ground-truth classes.
• Figure S1: Per-class top-1 zero-shot accuracy on the nuScenes-triplets validation set, with class frequencies shown in blue. Compared are ULIP-2 (Objaverse), ULIP-2 (nuScenes), and BlendCLIP (ours).