Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiCLIP: Domain Canonicalization via Structured Geometric Transformation

Pranav Mantini, Shishir K. Shah

TL;DR

This work hypothesizes that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors, and introduces BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment.

Abstract

Recent advances in vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities, yet adapting these models to specialized domains remains a significant challenge. Building on recent theoretical insights suggesting that independently trained VLMs are related by a canonical transformation, we extend this understanding to the concept of domains. We hypothesize that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors. Few-shot classification provides a natural setting for this alignment, as the limited labeled samples serve as the anchors required to estimate this transformation. Motivated by this hypothesis, we introduce BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment. Our approach is characterized by its extreme simplicity and low parameter footprint. Extensive evaluations across 11 standard benchmarks, including EuroSAT, DTD, and FGVCAircraft, demonstrate that BiCLIP consistently achieves state-of-the-art results. Furthermore, we provide empirical verification of existing geometric findings by analyzing the orthogonality and angular distribution of the learned transformations, confirming that structured alignment is the key to robust domain adaptation. Code is available at https://github.com/QuantitativeImagingLaboratory/BilinearCLIP

BiCLIP: Domain Canonicalization via Structured Geometric Transformation

TL;DR

This work hypothesizes that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors, and introduces BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment.

Abstract

Recent advances in vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities, yet adapting these models to specialized domains remains a significant challenge. Building on recent theoretical insights suggesting that independently trained VLMs are related by a canonical transformation, we extend this understanding to the concept of domains. We hypothesize that image features across disparate domains are related by a canonicalized geometric transformation that can be recovered using a small set of anchors. Few-shot classification provides a natural setting for this alignment, as the limited labeled samples serve as the anchors required to estimate this transformation. Motivated by this hypothesis, we introduce BiCLIP, a framework that applies a targeted transformation to multimodal features to enhance cross-modal alignment. Our approach is characterized by its extreme simplicity and low parameter footprint. Extensive evaluations across 11 standard benchmarks, including EuroSAT, DTD, and FGVCAircraft, demonstrate that BiCLIP consistently achieves state-of-the-art results. Furthermore, we provide empirical verification of existing geometric findings by analyzing the orthogonality and angular distribution of the learned transformations, confirming that structured alignment is the key to robust domain adaptation. Code is available at https://github.com/QuantitativeImagingLaboratory/BilinearCLIP
Paper Structure (22 sections, 8 equations, 3 figures, 4 tables)

This paper contains 22 sections, 8 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adaptation of Vision-Language Models
  4. Geometric Representation and Multimodal Alignment
  5. Preliminaries
  6. Contrastive Language-Image Pre-training Models (CLIP)
  7. Sigmoid Loss for Language-Image Pre-training (SigLIP)
  8. Shortcoming of the zero-shot CLIP
  9. BiCLIP: Structured Bilinear Alignment
  10. BiCLIP Theory
  11. Upper Triangular Structured Constraint
  12. BiCLIP: Bilinear Adaptation for CLIP
  13. BiSigLIP: Bilinear Adaptation for SigLIP
  14. Experimental Methodology
  15. Datasets
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Quantitative analysis of angular distribution on DTD dtd dataset. The zero-shot CLIP (a) shows significant overlap between positive and negative pairs. A structured geometric transformation on the image features (b) reduces the overlap area significantly.
  • Figure 2: The BiCLIP Adaptation Framework. Unlike standard CLIP which relies on a fixed dot product, BiCLIP introduces a trainable, structured transformation matrix $\mathbf{W}$ between the image and text modalities.
  • Figure 3: Few-shot performance comparison on various datasets. Our methods BiCLIP (black) and BiSigLIP (red) significantly outperform existing prompt tuning baselines across 1, 2, 4, 8, and 16 shots.