MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique

TL;DR

MM-Critic presents a holistic benchmark to evaluate multimodal critique in Large Multimodal Models across 8 task categories and 4471 samples, covering basic, correction, and comparative critique. It couples scalar metrics with textual Critique Scores and uses ground-truth grounded answers plus reference critiques anchored at 8 to stabilize judgments and improve reliability. The study reveals scaling trends, with closed-source models often outperforming open ones and a practical ~30B parameter threshold for strong critique performance, while highlighting challenges in correction and medium-quality cases. By providing a rigorous, multi-dimensional evaluation framework and open-source resources, MM-Critic advances the standardized assessment of LMM critique and supports development of more explainable and trustworthy multimodal systems.

Abstract

The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.

Figures (14)

• Figure 1: Multi-dimensional critique evaluation in MM-Critic. Basic critique includes binary correctness and textual feedback (Critique Accuracy, Critique Score); correction and comparative critique correspond to Correction Critique Score and Preference Accuracy, respectively.
• Figure 2: Scaling law on $\mathrm{ACC_{critic}}$ across models. Note that the parameter sizes of all closed-source LMMs are estimated, as their exact values are not publicly available. However, the relative scale among them is preserved — for example, Gemini-2.5-flash is known to be smaller than Gemini-2.5-pro.
• Figure 3:
• Figure 4: The relationship between the average length of textual critiques and critique scores across models.
• Figure 5: Basic Reference Critique Generation Prompt for GPT-4o.