Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection
Saroj Basnet, Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanoji, Marcos Zampieri
TL;DR
This work benchmarks seven open-source vision-language models on multimodal sarcasm detection across zero-, one-, and few-shot in-context prompting, evaluating both binary classification and explanation generation without fine-tuning. Using MuSE for explanations and MMSD2.0/SarcNet for multilingual image-caption data, the study reveals that instruction-tuned models like Gemma3 and InstructBLIP excel at detection, while LLaVA leads in generating visually grounded explanations; few-shot prompts offer limited gains. The results highlight a clear divergence between classification accuracy and explainability, underscoring the need for task-specific training or architectural adaptations to jointly optimize reasoning and rationale generation. The findings inform future directions in designing VLMs for nuanced ASD tasks and emphasize robust evaluation frameworks that couple detection with human-aligned explanations.
Abstract
Recent advances in open-source vision-language models (VLMs) offer new opportunities for understanding complex and subjective multimodal phenomena such as sarcasm. In this work, we evaluate seven state-of-the-art VLMs - BLIP2, InstructBLIP, OpenFlamingo, LLaVA, PaliGemma, Gemma3, and Qwen-VL - on their ability to detect multimodal sarcasm using zero-, one-, and few-shot prompting. Furthermore, we evaluate the models' capabilities in generating explanations to sarcastic instances. We evaluate the capabilities of VLMs on three benchmark sarcasm datasets (Muse, MMSD2.0, and SarcNet). Our primary objectives are twofold: (1) to quantify each model's performance in detecting sarcastic image-caption pairs, and (2) to assess their ability to generate human-quality explanations that highlight the visual-textual incongruities driving sarcasm. Our results indicate that, while current models achieve moderate success in binary sarcasm detection, they are still not able to generate high-quality explanations without task-specific finetuning.