CyCLIP: Cyclic Contrastive Language-Image Pretraining

TL;DR

CyCLIP tackles inconsistencies observed when CLIP-style contrastive learning jointly embeds images and text. By adding two cycle-consistency regularizers, CyCLIP enforces both cross-modal and in-modal geometric coherence, improving the alignment of image-text pairs while stabilizing modality-specific reasoning. Empirically, CyCLIP achieves notable gains in zero-shot classification and robustness to natural distribution shifts, and analysis indicates improved entity-level knowledge and a more favorable balance of alignment and uniformity on the embedding hypersphere. These results demonstrate that explicit geometric regularization can enhance the interoperability of multimodal representations for downstream tasks.

Abstract

Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. Such models typically require joint reasoning in the image and text representation spaces for downstream inference tasks. Contrary to prior beliefs, we demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. To mitigate this issue, we formalize consistency and propose CyCLIP, a framework for contrastive representation learning that explicitly optimizes for the learned representations to be geometrically consistent in the image and text space. In particular, we show that consistent representations can be learned by explicitly symmetrizing (a) the similarity between the two mismatched image-text pairs (cross-modal consistency); and (b) the similarity between the image-image pair and the text-text pair (in-modal consistency). Empirically, we show that the improved consistency in CyCLIP translates to significant gains over CLIP, with gains ranging from 10%-24% for zero-shot classification accuracy on standard benchmarks (CIFAR-10, CIFAR-100, ImageNet1K) and 10%-27% for robustness to various natural distribution shifts. The code is available at https://github.com/goel-shashank/CyCLIP.

Figures (6)

• Figure 1: An illustration of the planar geometry of the learned representations of image-text pairs by (a) CLIP and (b) CyCLIP. The edges indicate the distance between the representations i.e., $d({e}_{1}, {e}_{2}) = 1 - {\langle}{e}_{1}, {e}_{2}{\rangle}$, where $\langle \cdot , \cdot \rangle$ is the inner product. CyCLIP is cyclic consistent between image-text pairs as the in-modal distances, $d({T}_{\text{cat}},{T}_{\text{dog}}) \sim d({I}_{\text{cat}}, {I}_{\text{dog}})$, and the cross-modal distances, $d({T}_{\text{cat}}, {I}_{\text{dog}}) \sim d({I}_{\text{cat}}, {T}_{\text{dog}})$, are similar to each other unlike CLIP. Due to explicit consistency constraints, the test image of a cat is classified as a cat in the image as well as the text space.
• Figure 2: Illustrative overview for CyCLIP ($N=2$). It consists of 3 major components: (a) cross-modal contrastive alignment, (b) cross-modal consistency, and (c) in-modal consistency. Only (a) is present in CLIP, whereas our proposed regularizers in (b) and (c) mitigate inconsistency.
• Figure 3: The gap between the performances of CLIP and CyCLIP is much larger in coarse-grained scenario highlighting better entity-level knowledge representation in CyCLIP.
• Figure 4: Effect of varying the training dataset size on (a) Classification accuracy on ImageNet1K and (b) Effective Robustness on ImageNetV2.
• Figure 5: Distribution of superclasses across different number of subclasses. ImageNet1K/V2/Sketch and ImageNet-A/R have 1000 and 200 subclasses respectively.