Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding
Seyyed Saeid Cheshmi, Hahnemann Ortiz, James Mooney, Dongyeop Kang
TL;DR
This work tackles multimodal figurative language understanding by proposing a three-stage reasoning framework that distills chain-of-thought from a large teacher into a small student, followed by reinforcement learning with verifiable rewards. The approach enables explicit, inspectable reasoning across four figurative styles (sarcasm, humor, offense, metaphor) and supports cross-style transfer, with a unified training regime yielding a generalized model that rivals larger vision-language models. Key findings show that CoT reasoning substantially improves performance, cross-style transfer is strongest between related styles, and a combined training across styles delivers robust generalization while maintaining efficiency. The framework offers a scalable paradigm for interpretable multimodal reasoning with potential applicability to other complex tasks beyond figurative language.
Abstract
Vision-language models (VLMs) have demonstrated strong reasoning abilities in literal multimodal tasks such as visual mathematics and science question answering. However, figurative language, such as sarcasm, humor, and metaphor, remains a significant challenge, as it conveys intent and emotion through subtle incongruities between expressed and intended meanings. In multimodal settings, accompanying images can amplify or invert textual meaning, demanding models that reason across modalities and account for subjectivity. We propose a three-step framework for developing efficient multimodal reasoning models that can (i) interpret multimodal figurative language, (ii) provide transparent reasoning traces, and (iii) generalize across multiple figurative styles. Experiments across four styles show that (1) incorporating reasoning traces substantially improves multimodal figurative understanding, (2) reasoning learned in one style can transfer to others, especially between related styles like sarcasm and humor, and (3) training jointly across styles yields a generalized reasoning VLM that outperforms much larger open- and closed-source models. Our findings show that lightweight VLMs with verifiable reasoning achieve robust cross-style generalization while providing inspectable reasoning traces for multimodal tasks. The code and implementation are available at https://github.com/scheshmi/CrossStyle-MMR.