Towards Faithful Reasoning in Comics for Small MLLMs

TL;DR

This work identifies a paradox where naive Chain-of-Thought prompting can degrade CVQA performance for small multimodal models due to entanglement, irrelevant transitions, and inefficient exploration. It proposes MoCoT, a plan-execute-verify framework with typed subgoals, plus VERA, a structured reward for reinforcement fine-tuning, and GRPO to align outputs with faithful reasoning. The approach yields state-of-the-art results across five humor-centric and abstract CVQA benchmarks, with notable gains for compact models and demonstrated generalization to different backbones. By emphasizing trajectory-level faithfulness and modular reasoning, the work provides practical methods to improve robustness and transferability in resource-constrained multimodal reasoning tasks. The study also introduces faithfulness metrics USR and CAS to evaluate grounding and consistency beyond mere accuracy.

Abstract

Comic-based visual question answering (CVQA) poses distinct challenges to multimodal large language models (MLLMs) due to its reliance on symbolic abstraction, narrative logic, and humor, which differ from conventional VQA tasks. Although Chain-of-Thought (CoT) prompting is widely used to enhance MLLM reasoning, surprisingly, its direct application to CVQA often degrades performance, especially in small-scale models. Our theoretical and empirical analyses reveal that standard CoT in CVQA suffers from state entanglement, spurious transitions, and exploration inefficiency, with small models particularly vulnerable in resource-constrained settings. To address these issues, we propose a novel comic reasoning framework, designed to produce more faithful and transferable reasoning chains in small MLLMs. Specifically, our framework combines modular CoT generation with GRPO-based reinforcement fine-tuning and a novel structured reward. Beyond comic VQA, we further evaluate our approach on a broader class of humor-centric and abstract visual reasoning tasks, including meme understanding and editorial cartoon interpretation. Across five challenging benchmarks, our 3B model outperforms state-of-the-art methods, and plug-in experiments yield an additional average improvement of $\mathbf{12.1\%}$ across different MLLMs.

Figures (6)

• Figure 1: (A) Accuracy change with CoT prompting on CII-Bench, where naive CoT consistently degrades performance, with small MLLMs suffering larger drops and greater instability. The complete numerical results are provided in Appendix \ref{['app:cii-bench-full']}. (B) Our plug-in consistently improves accuracy across small MLLMs on DeepEval, compared with both w/ CoT and w/o CoT baselines.
• Figure 2: Representative failure cases of Qwen2.5-VL-3B (shown as Qwen in the figure) under naive CoT prompting. Typical errors include (A) satirical target confusion, (B) symbolic misalignment, and (C) salient cue omission, which directly lead to performance degradation. Our approach mitigates all the three factors.
• Figure 3: Our proposed MoCoT pipeline decomposes comic-based VQA tasks into structured sub-questions and sub-answers, followed by reflective reasoning and meta-level verification to guide final answer selection.
• Figure 4: (A) Overview of GRPO with our proposed VERA reward function. Given a prompt, the policy model generates multiple outputs, which are scored by the VERA reward model. Rewards are normalized into group-relative advantages, and KL regularization ensures stability with respect to the reference model. (B) Reduction of representative failure patterns under our framework.
• Figure 5: Sensitivity analysis of VERA reward weights under $\pm20\%$ perturbations.

Theorems & Definitions (21)

• Proposition 2.1: Limitations of Naive CoT
• Remark
• Definition 2.2: Weak Subgoal Coupling
• Proposition 2.3: Value Decomposition of MoCoT
• Remark
• Definition 2.4: VERA Reward
• Remark
• Lemma A.1: State entanglement is generic
• proof
• Lemma A.2: Inevitable spurious transitions