Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddings

TL;DR

The paper tackles few-shot adaptation under distribution shifts by proposing MixPro, which constructs a large mixed-embedding dataset by linearly combining source embeddings with a small set of target embeddings and trains a linear probe on these mixtures. The authors provide theoretical evidence that MixPro can outperform projection-based baselines like Pro2, especially under domain generalization and subpopulation shift regimes, and they validate the method empirically across eight datasets using 2–16 target samples per class. The results show MixPro achieving consistent gains, up to about 7%, and demonstrating robustness to hyperparameter settings via cross-validation with limited target data. Overall, MixPro offers a lightweight, data-efficient approach for last-layer adaptation that leverages both abundant source data and scarce target data to mitigate overfitting and improve target-domain generalization.

Abstract

Pretrained machine learning models need to be adapted to distribution shifts when deployed in new target environments. When obtaining labeled data from the target distribution is expensive, few-shot adaptation with only a few examples from the target distribution becomes essential. In this work, we propose MixPro, a lightweight and highly data-efficient approach for few-shot adaptation. MixPro first generates a relatively large dataset by mixing (linearly combining) pre-trained embeddings of large source data with those of the few target examples. This process preserves important features of both source and target distributions, while mitigating the specific noise in the small target data. Then, it trains a linear classifier on the mixed embeddings to effectively adapts the model to the target distribution without overfitting the small target data. Theoretically, we demonstrate the advantages of MixPro over previous methods. Our experiments, conducted across various model architectures on 8 datasets featuring different types of distribution shifts, reveal that MixPro can outperform baselines by up to 7\%, with only 2-4 target examples.

Figures (9)

• Figure 1: Comparison between methods on synthetic data.
• Figure 2: A trade-off between learning target information and preventing overfitting noise.Top: incorporating more target information with increasing $s$. The orange and blue dots represent examples from two classes in the spurious and core coordinates. The black line represents the decision boundary of the learned model along these first two coordinates. Bottom: With increasing $s$, the model has a larger component in the coordinates for random noise. The normalized weights vector is displayed in blue. Its component in the first two coordinates is shown in green, and the component in the directions of noise is shown in red.
• Figure 3: Left: A larger $p_\text{spu}$, indicating a more severe shift, necessitates a larger $s$; Right: A larger $r=\frac{\sigma_\xi^2}{m}$, signifying greater noise or a smaller target data size, necessitates a smaller $s$.
• Figure 4: Test accuracy on the target distribution versus target data size for all baselines across the 8 datasets we consider. Here, we use the SWAG-pretrained ViT-L/16 model as the backbone . Overall, while some methods may perform comparably to MixPro on certain datasets, they falter on others. In contrast, MixPro consistently achieves the best performance across datasets and data sizes.
• Figure 5: Results with hyperparameters tuned using cross-validation with only the few given target data. We use the SWAG-pretrained ViT-L/16 model as the backbone. In these scenarios, MixPro continues to outperform the others as the best method.

Theorems & Definitions (9)

• Theorem 5.1
• Theorem 5.3
• Proposition 1.1
• Corollary 1.2
• proof
• Lemma 1.3
• proof
• Lemma 1.4
• proof