Exploring Cross-Modal Flows for Few-Shot Learning

TL;DR

This paper proposes the first model-agnostic multi-step adjustment approach by learning a cross-modal velocity field: Flow Matching Alignment (FMA), and proposes an early-stopping solver, which terminates the transformation process earlier, improving both efficiency and accuracy.

Abstract

Aligning features from different modalities, is one of the most fundamental challenges for cross-modal tasks. Although pre-trained vision-language models can achieve a general alignment between image and text, they often require parameter-efficient fine-tuning (PEFT) for further adjustment. Today's PEFT methods (e.g., prompt tuning, LoRA-based, or adapter-based) always selectively fine-tune a subset of parameters, which can slightly adjust either visual or textual features, and avoid overfitting. In this paper, we are the first to highlight that all existing PEFT methods perform one-step adjustment. It is insufficient for complex (or difficult) datasets, where features of different modalities are highly entangled. To this end, we propose the first model-agnostic multi-step adjustment approach by learning a cross-modal velocity field: Flow Matching Alignment (FMA). Specifically, to ensure the correspondence between categories during training, we first utilize a fixed coupling strategy. Then, we propose a noise augmentation strategy to alleviate the data scarcity issue. Finally, we design an early-stopping solver, which terminates the transformation process earlier, improving both efficiency and accuracy. Compared with one-step PEFT methods, FMA has the multi-step rectification ability to achieve more precise and robust alignment. Extensive results have demonstrated that FMA can consistently yield significant performance gains across various benchmarks and backbones, particularly on challenging datasets.

Figures (7)

• Figure 1: Comparisons of the cross-modal alignment process of different methods. (a) The overview pipeline of CLIP radford2021learning for zero-shot cross-modal alignment and classification. Some image features and their corresponding text features are not well-aligned. (b-d) The alignment process of three typical types of state-of-the-art PEFT approaches, which adjust image or text features in one single step. The arrow shows the adjustment of corresponding features during the adaptation. For difficult classes, the image features still may be far from the corresponding text features by one step adjustment. (e) FMA achieves multi-step cross-modal alignment, which succeeds in aligning text-image features for difficult classes.
• Figure 2: Performance of CoOp and linear probing. We experimented with the 16-shot setting and chose CLIP RN50 as the backbone.
• Figure 3: Explanation of the two challenges. (a) A flow matching example to transform image features (red) to text features (blue) distribution. Each distribution consists of features of three categories. (b) Two examples of how a trained velocity transforms an image feature into a text feature using the vanilla flow matching inference strategy.
• Figure 4: Overview of Flow Matching Alignment (FMA). The main idea of the training stage is to learn a velocity field, which can transform image features to corresponding text features. Two designs are proposed: coupling enhancement and noise augmentation. During the inference, FMA applies an early-stopping solver that can output intermediate features for classification.
• Figure 5: (a) Red line: Average distance to target figure at different timesteps. Black line: Accuracy using features at different timesteps for classification. (b) At different timesteps, the distance between the intermediate features and the correct/incorrect text feature.