Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

Davide Bucciarelli, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

TL;DR

This work investigates whether Multimodal LLMs can substitute traditional image captioning systems by evaluating zero-shot performance and domain adaptation through parameter-efficient fine-tuning. It uses LLaVA-based models with a CLIP visual encoder and tests several PEFT methods—Prompt Learning, Prefix Tuning, LoRA, and DoRA—alongside full fine-tuning. The findings reveal strong zero-shot capabilities but limited domain adaptation when fine-tuning aggressively; PEFT approaches offer favorable trade-offs between caption quality and generalization, whereas full fine-tuning can harm cross-domain performance. The results inform future development of more robust, concise, and hallucination-resistant multimodal captioning systems that generalize across diverse domains.

Abstract

The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs.

Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

TL;DR

This work investigates whether Multimodal LLMs can substitute traditional image captioning systems by evaluating zero-shot performance and domain adaptation through parameter-efficient fine-tuning. It uses LLaVA-based models with a CLIP visual encoder and tests several PEFT methods—Prompt Learning, Prefix Tuning, LoRA, and DoRA—alongside full fine-tuning. The findings reveal strong zero-shot capabilities but limited domain adaptation when fine-tuning aggressively; PEFT approaches offer favorable trade-offs between caption quality and generalization, whereas full fine-tuning can harm cross-domain performance. The results inform future development of more robust, concise, and hallucination-resistant multimodal captioning systems that generalize across diverse domains.

Abstract

The task of image captioning demands an algorithm to generate natural language descriptions of visual inputs. Recent advancements have seen a convergence between image captioning research and the development of Large Language Models (LLMs) and Multimodal LLMs -- like GPT-4V and Gemini -- which extend the capabilities of text-only LLMs to multiple modalities. This paper investigates whether Multimodal LLMs can supplant traditional image captioning networks by evaluating their performance on various image description benchmarks. We explore both the zero-shot capabilities of these models and their adaptability to different semantic domains through fine-tuning methods, including prompt learning, prefix tuning, and low-rank adaptation. Our results demonstrate that while Multimodal LLMs achieve impressive zero-shot performance, fine-tuning for specific domains while maintaining their generalization capabilities intact remains challenging. We discuss the implications of these findings for future research in image captioning and the development of more adaptable Multimodal LLMs.

Paper Structure

This paper contains 11 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Preliminaries
  5. Personalization Strategies
  6. Experimental Evaluation
  7. Datasets and Evaluation Metrics
  8. Implementation and Training Details
  9. Experimental Results
  10. Computational Analysis
  11. Conclusion

Figures (4)

  • Figure 1: Overview of our approach. We investigate whether Multimodal LLMs can supplant traditional captioners by assessing their adaptability to different semantic domains and through the usage of different adaptation techniques.
  • Figure 2: Comparison between CIDEr scores achieved by the different versions of LLaVA-v1.5 (first row) and by those of LLaVA-v1.6 (second row) on out-of-domain datasets including nocaps, VizWiz, and TextCaps.
  • Figure 3: Sample image descriptions generated by the zero-shot MLLM in comparison with those generated by the MLLM fine-tuned with prompt learning. For reference, we also report a ground-truth caption (GT) associated to each image.
  • Figure 4: Qualitative results on sample images from nocaps (first row), ViZWiz (second row), and TextCaps (third row). We compare captions predicted by the CLIP-Captioner model barraco2022unreasonable, the MLLM after full fine-tuning, and the MLLM after prompt learning.