Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks

Jiayu Wang, Yang Jiao, Yue Yu, Tianwen Qian, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang

TL;DR

OmniGenBench introduces a comprehensive benchmark with 57 sub-tasks spanning perception-centric and cognition-centric image generation to assess instruction-following in large multimodal models. It employs a dual-mode evaluation—automated perception parsing for appearance/dynamics tasks and LLM-based judgment with task-specific prompts for cognitive tasks—and defines OmniScore as $OmniScore = 0.8 × Consistency + 0.1 × Realism + 0.1 × Aesthetics$ to quantify cross-task performance. The authors validate OmniGenBench by benchmarking mainstream models, finding GPT-4o-Native to achieve state-of-the-art performance across most dimensions, with insights into strengths and limitations of rivals and open-source counterparts. The work provides a principled, reproducible framework for evaluating general-purpose multimodal generation and guiding future improvements in model capabilities and evaluation methodologies.

Abstract

Recent breakthroughs in large multimodal models (LMMs), such as the impressive GPT-4o-Native, have demonstrated remarkable proficiency in following general-purpose instructions for image generation. However, current benchmarks often lack the necessary breadth and depth to fully evaluate the diverse capabilities of these models. To overcome this limitation, we introduce OmniGenBench, a novel and comprehensive benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs across both perception-centric and cognition-centric dimensions. Our OmniGenBench includes 57 diverse sub-tasks grounded in real-world scenarios, systematically categorized according to the specific model capabilities they demand. For rigorous evaluation, we further employ a dual-mode protocol. This protocol utilizes off-the-shelf visual parsing tools for perception-centric tasks and a powerful LLM-based judger for cognition-centric tasks to assess the alignment between generated images and user instructions. Using OmniGenBench, we evaluate mainstream generative models, including prevalent models like GPT-4o, Gemini-2.0-Flash, and Seedream, and provide in-depth comparisons and analyses of their performance.Code and data are available at https://github.com/emilia113/OmniGenBench.

OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks

TL;DR

OmniGenBench introduces a comprehensive benchmark with 57 sub-tasks spanning perception-centric and cognition-centric image generation to assess instruction-following in large multimodal models. It employs a dual-mode evaluation—automated perception parsing for appearance/dynamics tasks and LLM-based judgment with task-specific prompts for cognitive tasks—and defines OmniScore as to quantify cross-task performance. The authors validate OmniGenBench by benchmarking mainstream models, finding GPT-4o-Native to achieve state-of-the-art performance across most dimensions, with insights into strengths and limitations of rivals and open-source counterparts. The work provides a principled, reproducible framework for evaluating general-purpose multimodal generation and guiding future improvements in model capabilities and evaluation methodologies.

Abstract

Recent breakthroughs in large multimodal models (LMMs), such as the impressive GPT-4o-Native, have demonstrated remarkable proficiency in following general-purpose instructions for image generation. However, current benchmarks often lack the necessary breadth and depth to fully evaluate the diverse capabilities of these models. To overcome this limitation, we introduce OmniGenBench, a novel and comprehensive benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs across both perception-centric and cognition-centric dimensions. Our OmniGenBench includes 57 diverse sub-tasks grounded in real-world scenarios, systematically categorized according to the specific model capabilities they demand. For rigorous evaluation, we further employ a dual-mode protocol. This protocol utilizes off-the-shelf visual parsing tools for perception-centric tasks and a powerful LLM-based judger for cognition-centric tasks to assess the alignment between generated images and user instructions. Using OmniGenBench, we evaluate mainstream generative models, including prevalent models like GPT-4o, Gemini-2.0-Flash, and Seedream, and provide in-depth comparisons and analyses of their performance.Code and data are available at https://github.com/emilia113/OmniGenBench.

Paper Structure

This paper contains 17 sections, 1 equation, 19 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Generation Models
  4. Image Generation Benchmarks
  5. OmniGenBench
  6. Task Coverage
  7. Perception-Centric Generation
  8. Cognition-Centric Generation
  9. Data Construction
  10. Evaluation Protocol
  11. Experiments
  12. Evaluation of Mainstream Approaches
  13. Agreement with Human Evaluation
  14. In-depth Qualitative Comparison
  15. Conclusion
  16. ...and 2 more sections

Figures (19)

  • Figure 1: Evaluation of leading generative models in handling the general-purpose instructional task.
  • Figure 2: The proposed taxonomy of OmniGenBench, accompanied by representative tasks sampled from across different categories.
  • Figure 3: The data curation pipeline of our OmniGenBench.
  • Figure 4: Illustration of the evaluation pipeline of our OmniGenBench.
  • Figure 5: Results of agreement rate between our OmniScore and human evaluation.
  • ...and 14 more figures