MultiModal Fine-tuning with Synthetic Captions

TL;DR

This work tackles the gap between multimodal pre-training and unimodal fine-tuning by converting unimodal image datasets into multimodal ones through synthetic captions generated by Multimodal LLMs. It introduces a supervised contrastive fine-tuning objective that leverages class labels and a novel inference strategy that averages text embeddings across multiple captions per class. The proposed approach, including synthetic dataset generation, a class-aware loss with a balanced combination of standard and supervised components, and class-prototype text embeddings, yields consistent improvements across 13 datasets, with notable gains in few-shot settings and even no-training scenarios. The results demonstrate that rich synthetic captions can effectively transfer the strengths of multimodal pre-training to downstream tasks, offering a practical data-centric paradigm for bridging multimodal pre-training and fine-tuning, with open-source code available at the provided repository.

Abstract

In this paper, we address a fundamental gap between pre-training and fine-tuning of deep neural networks: while pre-training has shifted from unimodal to multimodal learning with enhanced visual understanding, fine-tuning predominantly remains unimodal, limiting the benefits of rich pre-trained representations. To bridge this gap, we propose a novel approach that transforms unimodal datasets into multimodal ones using Multimodal Large Language Models (MLLMs) to generate synthetic image captions for fine-tuning models with a multimodal objective. Our method employs carefully designed prompts incorporating class labels and domain context to produce high-quality captions tailored for classification tasks. Furthermore, we introduce a supervised contrastive loss function that explicitly encourages clustering of same-class representations during fine-tuning, along with a new inference technique that leverages class-averaged text embeddings from multiple synthetic captions per image. Extensive experiments across 13 image classification benchmarks demonstrate that our approach outperforms baseline methods, with particularly significant improvements in few-shot learning scenarios. Our work establishes a new paradigm for dataset enhancement that effectively bridges the gap between multimodal pre-training and fine-tuning. Our code is available at https://github.com/s-enmt/MMFT.

Figures (11)

• Figure 1: Overview of our proposed approach. Our method consists of three key components: (a) Synthetic multimodal dataset generation using MLLM, (b) Supervised contrastive fine-tuning that leverages both image-caption pairs and class information, and (c) Class-averaged text embedding inference.
• Figure 2: Average classification accuracy ($\%$) of ResNet-50 across 13 datasets with different numbers of training examples per class (shots). Ours uses multiple captions focusing on three different characteristics: visual, shape, and texture, while Ours(SingleCap) uses a single caption focusing only on the visual characteristic.
• Figure 3: Comparison of average classification accuracy ($\%$) with ResNet-50 between fine-tuned approaches and caption-based approaches without training across 8 datasets.
• Figure 4: Impact of supervised contrastive loss weight $w$ on average classification accuracy ($\%$) across 12 datasets (excluding ImageNet).
• Figure 5: Results of few-shot learning for each dataset.