Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

TL;DR

This work systematically investigates how In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting affect the linguistic grounding abilities of Multimodal Large Language Models (MLLMs) on the VALSE benchmark. By evaluating 14 diverse models across zero-shot and few-shot settings and employing MMICES-based demonstration selection and CoT-generated reasoning, the study reveals that ICL and CoT substantially enhance performance in tasks requiring intermediate reasoning. A key finding is that captioning-pretrained MLLMs tend to excel in zero-shot scenarios, while models trained on interleaved image-text data gain more from few-shot prompts, especially when demonstrations are textually similar to the query pair. The results underscore the importance of pretraining data composition and prompting strategies for robust visio-linguistic grounding, with practical implications for deploying efficient, reasoning-capable MLLMs without extensive fine-tuning.

Abstract

The linguistic capabilities of Multimodal Large Language Models (MLLMs) are critical for their effective application across diverse tasks. This study aims to evaluate the performance of MLLMs on the VALSE benchmark, focusing on the efficacy of few-shot In-Context Learning (ICL), and Chain-of-Thought (CoT) prompting. We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets. The experimental results reveal that ICL and CoT prompting significantly boost model performance, particularly in tasks requiring complex reasoning and contextual understanding. Models pretrained on captioning datasets show superior zero-shot performance, while those trained on interleaved image-text data benefit from few-shot learning. Our findings provide valuable insights into optimizing MLLMs for better grounding of language in visual contexts, highlighting the importance of the composition of pretraining data and the potential of few-shot learning strategies to improve the reasoning abilities of MLLMs.

Figures (17)

• Figure 1: Zero-Shot, Few-Shot ICL, and Few-Shot ICL+CoT Evaluation Comparison on the Relations Task. In the Zero-Shot approach, the model incorrectly responds to the question. Few-Shot ICL, using prior examples, correctly identifies the horse behind a wooden fence. Few-Shot ICL+CoT, which is beneficial for tasks requiring intermediate reasoning steps, e.g. counting, relational understanding, and coreference resolution, also correctly identifies the horse by employing a detailed step-by-step reasoning process.
• Figure 2: Sample instances from the VALSE benchmark parcalabescu-etal-2022-valse.
• Figure 3: Sample data demonstrating the differences between image-text pairs, and interleaved text and image data used in training MLLMs.
• Figure 4: Example model predictions on instances from the Existence task, with demonstrations selected based on both visual and textual similarity (setting S).
• Figure 5: Example model predictions on instances from the Plurality task, with demonstrations selected based on both visual and textual similarity (setting S).