GenExam: A Multidisciplinary Text-to-Image Exam

TL;DR

GenExam introduces the first multidisciplinary text-to-image exam benchmark, featuring 1,000 prompts across 10 subjects, ground-truth images, and a four-level taxonomy to evaluate semantic correctness and visual plausibility. Prompts and scoring points are co-designed by GPT-5 and human reviewers, and evaluation uses a two-dimensional scheme combining semantic accuracy with visual plausibility, producing strict and relaxed scores. Across 18 baselines, state-of-the-art closed-source models achieve only modest strict performance (all <15%), while open-source models lag behind, revealing a large gap between model families and the challenge of disciplined, knowledge-rich image generation. The benchmark provides data and an evaluation framework to drive progress toward expert-level, multidisciplinary image generation and informs pathways toward general artificial intelligence.

Abstract

Exams are a fundamental test of expert-level intelligence and require integrated understanding, reasoning, and generation. Existing exam-style benchmarks mainly focus on understanding and reasoning tasks, and current generation benchmarks emphasize the illustration of world knowledge and visual concepts, neglecting the evaluation of rigorous drawing exams. We introduce GenExam, the first benchmark for multidisciplinary text-to-image exams, featuring 1,000 samples across 10 subjects with exam-style prompts organized under a four-level taxonomy. Each problem is equipped with ground-truth images and fine-grained scoring points to enable a precise evaluation of semantic correctness and visual plausibility. Experiments show that even state-of-the-art models such as GPT-Image-1 and Gemini-2.5-Flash-Image achieve less than 15% strict scores, and most models yield almost 0%, suggesting the great challenge of our benchmark. By framing image generation as an exam, GenExam offers a rigorous assessment of models' ability to integrate understanding, reasoning, and generation, providing insights on the path to general AGI. Our benchmark and evaluation code are released at https://github.com/OpenGVLab/GenExam.

Figures (14)

• Figure 1: Examples and performance of state-of-the-art text-to-image models on GenExam. Orange dashed circles indicate scoring points. GenExam contains complex and diverse prompts resembling human exams, and pose great challenge to existing models.
• Figure 1: Key statistics of GenExam.
• Figure 2: Examples from GenExam. GenExam contains 1,000 exam-style prompts that span over 10 subjects and corresponding reference images for multidisciplinary text-to-image exam.
• Figure 3: Mean absolute error (MAE) and correlations between human and automatic evaluation. Our evaluation shows strong alignment with human preferences, evidented by low MAE and high correlations.
• Figure 4: Data curation pipeline. We use pre-defined taxonomy to collect web images and existing datasets, and conduct annotating and filtering based on GPT-5 and manual check.