AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Models

TL;DR

AEGIS targets the core challenge of world-knowledge reasoning in Unified Multimodal Models by introducing a comprehensive, multi-task benchmark with 1,050 manually annotated questions across 21 topics and six reasoning types. It replaces fragile, score-based judgments with Deterministic Checklist-based Evaluation (DCE), enabling reliable, atomic yes/no judgments tied to reference reasoning. Across extensive experiments, most UMMs exhibit notable world-knowledge deficits, with performance degrading on complex reasoning, while simple plug-in reasoning modules offer partial mitigation. The work provides a principled framework for cross-task evaluation, diagnosing component bottlenecks and guiding future development toward more robust, knowledge-grounded multimodal systems.

Abstract

The capability of Unified Multimodal Models (UMMs) to apply world knowledge across diverse tasks remains a critical, unresolved challenge. Existing benchmarks fall short, offering only siloed, single-task evaluations with limited diagnostic power. To bridge this gap, we propose AEGIS (\emph{i.e.}, \textbf{A}ssessing \textbf{E}diting, \textbf{G}eneration, \textbf{I}nterpretation-Understanding for \textbf{S}uper-intelligence), a comprehensive multi-task benchmark covering visual understanding, generation, editing, and interleaved generation. AEGIS comprises 1,050 challenging, manually-annotated questions spanning 21 topics (including STEM, humanities, daily life, etc.) and 6 reasoning types. To concretely evaluate the performance of UMMs in world knowledge scope without ambiguous metrics, we further propose Deterministic Checklist-based Evaluation (DCE), a protocol that replaces ambiguous prompt-based scoring with atomic ``Y/N'' judgments, to enhance evaluation reliability. Our extensive experiments reveal that most UMMs exhibit severe world knowledge deficits and that performance degrades significantly with complex reasoning. Additionally, simple plug-in reasoning modules can partially mitigate these vulnerabilities, highlighting a promising direction for future research. These results highlight the importance of world-knowledge-based reasoning as a critical frontier for UMMs.

Figures (10)

• Figure 1: Illustration of AEGIS benchmark. The contribution of AEGIS includes 1) a comprehensive and challenging benchmark with four differenct visual understanding and generation tasks, covering a board knowledge aspect (i.e., world knowledge); 2) a deterministic checlist-based evaluation protocol for concrete evaluation results; 3) empirical analysis for state-of-the-art unified multimodal models and other generative models to reveal the vulnerability on world knowledge and reasoning.
• Figure 2: Data construction and evaluation pipeline of our proposed AEGIS. Based on the board taxonomy aspect from human-in-the-loop exploration, AEGIS features a high-quality data construction procedure, using human-in-the-loop exploration and optimal annotation methods (web, tool, or expert) to create reasoning-enhanced questions. Another key highlight is the novel deterministic checklist-based evaluation (DCE), where an MLLM first generates a checklist of atomic "Y/N" questions from the reference answer. A judge MLLM then uses this checklist to produce clear, concrete, and reliable judgments of the model's prediction.
• Figure 3: Examples of AEGIS Benchmark, where red color indicates the key points examined in the question. AEGIS covers four visual generative tasks with 1,050 reasoning-enhanced questions from 21 different topics, which is useful to explore the generation capabilities of UMMs and other generative models under both broad knowledge aspects (i.e., world knowledge) and different reasoning types.
• Figure 4: Visualization of five state-of-the-art UMMs on Understanding, Generation, and Editing tasks. Green color means correct responses, and red color means wrong answers. Nana Banana expresses promising image generation and editing quality, and has better main object consistency in editing tasks than others. Meanwhile, open-sourced models don't perform well in these tasks.
• Figure 5: Visualization of failure cases with raw and LLM rewritten prompts. We highlight the keypoints in the answers by red color. Though external reasoning modules (e.g., Gemini) can ease the generation difficulty by rewritting complex prompts, there still exist gaps towards precise reasoning capabilities under diverse tasks across world knowledge aspects.