The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning

TL;DR

This survey reframes AI evaluation as a progression through cognitively demanding examinations, moving from the recognition-focused era (ImageNet/COCO) to reasoning (VQA, VQA-CP, GQA, VCR) and ultimately to expert-level, holistic benchmarks for Multimodal Large Language Models (MLLMs). It documents Level I through IV benchmarks, highlighting how each level uncovers specific weaknesses—shortcuts, binding failures, adversarial fragility, knowledge gaps, and embodied planning limitations—and discusses the rise of living benchmarks and adversarial pipelines to prevent stagnation. The work emphasizes that high benchmark scores often reflect test-taking acumen rather than transferable intelligence, advocating for diagnostic, process-oriented metrics (e.g., CoT rationales, per-dimension profiling, and tool-use diagnostics) and living regimes (continuous refresh, red-teaming, versioning). The authors argue that advancing AI requires benchmarks that test holistic, open-ended cognition—planning, embodied interaction, social intelligence, and creativity—while ensuring reproducibility and guarding against contaminations. In short, AI evaluation should itself evolve into a dynamic, adversarial enterprise that continually redefines our goals for true multimodal understanding and intelligent behavior.

Abstract

This survey paper chronicles the evolution of evaluation in multimodal artificial intelligence (AI), framing it as a progression of increasingly sophisticated "cognitive examinations." We argue that the field is undergoing a paradigm shift, moving from simple recognition tasks that test "what" a model sees, to complex reasoning benchmarks that probe "why" and "how" it understands. This evolution is driven by the saturation of older benchmarks, where high performance often masks fundamental weaknesses. We chart the journey from the foundational "knowledge tests" of the ImageNet era to the "applied logic and comprehension" exams such as GQA and Visual Commonsense Reasoning (VCR), which were designed specifically to diagnose systemic flaws such as shortcut learning and failures in compositional generalization. We then survey the current frontier of "expert-level integration" benchmarks (e.g., MMBench, SEED-Bench, MMMU) designed for today's powerful multimodal large language models (MLLMs), which increasingly evaluate the reasoning process itself. Finally, we explore the uncharted territories of evaluating abstract, creative, and social intelligence. We conclude that the narrative of AI evaluation is not merely a history of datasets, but a continuous, adversarial process of designing better examinations that, in turn, redefine our goals for creating truly intelligent systems.