Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd

TL;DR

Multimodal UNcommonsense (MUN) targets the brittleness of vision-language models when faced with uncommon or culturally-specific scenarios by introducing two tasks, MUN-vis and MUN-lang. It couples a human- and LLM-generated dataset with a retrieval-based in-context learning framework, employing a Multimodal Ensemble Retriever (MER) to retrieve semantically relevant textual and visual exemplars despite intentional cross-modal discordance. Across seven vision-language models, R-ICL yields consistent improvements in automatic and human evaluations, demonstrating enhanced abductive and contextual reasoning in atypical settings. The work provides a crucial benchmark and a scalable reasoning paradigm that advances robustness and adaptability of multimodal AI in real-world, culturally diverse contexts.

Abstract

Commonsense reasoning in multimodal contexts remains a foundational challenge in artificial intelligence. We introduce Multimodal UNcommonsense(MUN), a benchmark designed to evaluate models' ability to handle scenarios that deviate from typical visual or contextual expectations. MUN pairs visual scenes with surprising or unlikely outcomes described in natural language, prompting models to either rationalize seemingly odd images using everyday logic or uncover unexpected interpretations in ordinary scenes. To support this task, we propose a retrieval-based in-context learning (R-ICL) framework that transfers reasoning capabilities from larger models to smaller ones without additional training. Leveraging a novel Multimodal Ensemble Retriever (MER), our method identifies semantically relevant exemplars even when image and text pairs are deliberately discordant. Experiments show an average improvement of 8.3% over baseline ICL methods, highlighting the effectiveness of R-ICL in low-frequency, atypical settings. MUN opens new directions for evaluating and improving visual-language models' robustness and adaptability in real-world, culturally diverse, and non-prototypical scenarios.

Figures (17)

• Figure 1: Multimodal UNcommonsense Reasoning aims to produce explanations that make given outcomes appear likely. For example, overripe bananas (an uncommon context) can still be used for baking a sweet, moist banana cake (a common outcome), while a bag on a bench (common context) leads to an arrest (uncommon outcome). This highlights the challenge of bridging visual cues with logical reasoning, as addressed in our Multimodal Uncommonsense (MUN) dataset.
• Figure 2: MUN examples. The first two examples are from MUN-vis and the next two examples come from MUN-lang; explanations are written by human annotators. Note that textual context is only used during dataset generation.
• Figure 3: t-SNE visualization of MUN-vis (a) and MUN-lang (b) based SimCSE gao-etal-2021-simcse across categories.
• Figure 4: Explanation token length distributions in MUN: The left section represents MUN-vis, while the right section depicts MUN-lang, derived from calculations on the development sets of each data subset.
• Figure 5: The n-gram distribution entropies for MUN-vis (left) and MUN-lang (right) were calculated based on the development sets for each data subset.