Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization

TL;DR

This work tackles few-shot out-of-distribution generalization under two-dimensional distribution shifts by introducing Bayes-CAL, a Bayesian cross-modal alignment framework that fine-tunes only text representations. By disentangling category- from environment-related image information through a gradient orthogonal loss and enforcing environment-invariant predictions via IRM, Bayes-CAL leverages variational posteriors over task-specific parameters to reduce overfitting. The method achieves state-of-the-art OoD performance on NICO, CCD, PACS, and VLCS, with compelling base-to-new generalization and stable results across unseen classes. Empirical analyses, ablations, and optimization landscape visualizations illuminate why alignment learning, aided by Bayesian regularization, yields robust cross-domain generalization in few-shot settings.

Abstract

Recent advances in large pre-trained models showed promising results in few-shot learning. However, their generalization ability on two-dimensional Out-of-Distribution (OoD) data, i.e., correlation shift and diversity shift, has not been thoroughly investigated. Researches have shown that even with a significant amount of training data, few methods can achieve better performance than the standard empirical risk minimization method (ERM) in OoD generalization. This few-shot OoD generalization dilemma emerges as a challenging direction in deep neural network generalization research, where the performance suffers from overfitting on few-shot examples and OoD generalization errors. In this paper, leveraging a broader supervision source, we explore a novel Bayesian cross-modal image-text alignment learning method (Bayes-CAL) to address this issue. Specifically, the model is designed as only text representations are fine-tuned via a Bayesian modelling approach with gradient orthogonalization loss and invariant risk minimization (IRM) loss. The Bayesian approach is essentially introduced to avoid overfitting the base classes observed during training and improve generalization to broader unseen classes. The dedicated loss is introduced to achieve better image-text alignment by disentangling the causal and non-casual parts of image features. Numerical experiments demonstrate that Bayes-CAL achieved state-of-the-art OoD generalization performances on two-dimensional distribution shifts. Moreover, compared with CLIP-like models, Bayes-CAL yields more stable generalization performances on unseen classes. Our code is available at https://github.com/LinLLLL/BayesCAL.

Figures (5)

• Figure 1: Illustration of the proposed Bayesian cross-modal alignment learning paradigms. To avoid overfitting on the few-shot training samples, $\varphi^{(t)}$ is modelled by a variational distribution $q(\varphi^{(t)})$ to approximate its posterior distribution.
• Figure 2: (a) Pipeline of the proposed Bayes-CAL. The top and the bottom branches are used to generate category-related texts feature and environment-related texts feature, respectively. (b) The image-text alignment process. The cross-modal alignment is achieved by maximizing the cosine similarities between the image feature $f_I(X)$ and its corresponding texts feature $f_T(Y_{gt})$ (where $Y_{gt}$ is the ground-true class name of $X$). (c) Few-shot OoD generalization learning in downstream tasks. The embeddings of new classes "dog" and "bus" are very close to pre-training classes "cat" and "car" in the natural language models, respectively. While in the few-shot OoD setting, the image feature of the same category extracted by different environments may be various. This results in previous models hard to achieve image-text alignment since conditional information (especially for data dominated by complex correlation shift) can not be efficiently extracted from the image features. Benefiting from the image-text alignment incorporating the proposed loss, regardless of various environmental information, texts features can be aligned with the disentangled image features by a few tuning steps of texts representations. Note that the element positions of the category-related and environment-related features can be random.
• Figure 3: (a) Test accuracy on CCD at different epochs. The test accuracy of Bayes-CAL is more stable than CoOp as the training epoch increases. (b) Visualization of the optimization landscape. The x-axis and y-axis represent the first and the second principal components of the parameters, respectively. The red lines are the corresponding optimization trajectories. Without the pre-trained text encoder, Bayes-CAL(LV) and Bayes-CAL(W2V) still see much faster convergence and lower target loss compared to the conventional visual model.
• Figure 4: The three instances of the task-specific network in the Bayes-CAL and the task-specific network in the visual model.
• Figure 5: Visualization of the image features by 2D t-SNE. Categories belonging to animals and vehicles are represented by dots and crosses in different colors, respectively.