CLIMP: Contrastive Language-Image Mamba Pretraining

TL;DR

CLIP's ViT backbones incur quadratic complexity $O(L^2)$ and can rely on spurious correlations under distribution shifts. CLIMP replaces both vision and text encoders with Mamba state-space models, achieving linear complexity $O(L)$ and native resolution handling, while enabling dense captioning retrieval via autoregressive text encoding. The work reports state-of-the-art retrieval and robustness on CLIP benchmarks and ImageNet-O, with substantial memory and FLOPs savings at high resolutions and strong performance without bespoke position encoding schemes. It further analyzes the spatial inductive bias of VMamba, embedding geometry, and scaling behavior, supporting the viability of state-space models as robust, efficient alternatives to Transformers for vision-language pretraining. Overall, CLIMP demonstrates that fully Mamba-based vision-language pretraining can surpass Transformer-based CLIP in retrieval, robustness, high-resolution processing, and long-context capabilities.

Abstract

Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.

Figures (6)

• Figure 1: CLIP vs. CLIMP. By replacing Transformer encoders with Mamba-based models, CLIMP achieves sub-quadratic $O(L)$ complexity instead of quadratic $O(L^2)$, while removing the fixed resolution and 77-token text limitations inherent to standard CLIP.
• Figure 2: Visualization of Image-Text similarity for the caption: "A large porch with a wooden fence and no roof." CLIMP (similarity: 0.545) produces spatially coherent attention focused on the porch and fence, while RoPE-ViT (0.501) and FlexViT (0.452) show scattered, less interpretable patterns. Warmer colors indicate higher similarity.
• Figure 3: Efficiency analysis. CLIMP achieves superior memory and computational efficiency across all resolutions. (Left) Memory overhead is 4--57$\times$ lower. (Right) FLOPs scale linearly, yielding up to 1.8$\times$ reduction—a gap that widens with resolution.
• Figure 4: Scaling laws of CLIMP trained on CC dataset and evaluated on ImageNet-1K Acc@1. Performance improves consistently as training data scales from 1M to 12M samples.
• Figure 5: Visualization of image-text similarity. CLIMP exhibits more interpretable alignment and better similarity performance.