Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PixelArena: A benchmark for Pixel-Precision Visual Intelligence

Feng Liang, Sizhe Cheng, Chenqi Yi

TL;DR

PixelArena introduces a pixel-precision benchmark for evaluating fine-grained generative capabilities of multimodal LLMs by casting image generation as semantic mask production on CelebAMask-HQ and COCO. It demonstrates that Gemini 3 Pro Image exhibits emergent zero-shot segmentation skills, surpassing several baselines in qualitative and quantitative assessments while highlighting failure modes and potential data-contamination considerations. The study uses color-encoding prompts to map generated images to label masks, analyzes robustness via sampling variability and encoding shuffles, and explores performance on a harder dataset to illuminate limitations and future research directions. Overall, the work provides a novel, objective regime for assessing visual reasoning, generalization, and interpretability in multimodal generation systems, informing future benchmarks and multimodal curricula.

Abstract

Multi-modal large language models that have image output are emerging. Many image generation benchmarks focus on aesthetics instead of fine-grained generation capabilities. In PixelArena, we propose using semantic segmentation tasks to objectively examine their fine-grained generative intelligence with pixel precision. We find the latest Gemini 3 Pro Image has emergent image generation capabilities that generate semantic masks with high fidelity under zero-shot settings, showcasing visual intelligence unseen before and true generalization in new image generation tasks. We further investigate its results, compare them qualitatively and quantitatively with those of other models, and present failure cases. The findings not only signal exciting progress in the field but also provide insights into future research related to multimodality, reasoning, interpretability and benchmarking.

PixelArena: A benchmark for Pixel-Precision Visual Intelligence

TL;DR

PixelArena introduces a pixel-precision benchmark for evaluating fine-grained generative capabilities of multimodal LLMs by casting image generation as semantic mask production on CelebAMask-HQ and COCO. It demonstrates that Gemini 3 Pro Image exhibits emergent zero-shot segmentation skills, surpassing several baselines in qualitative and quantitative assessments while highlighting failure modes and potential data-contamination considerations. The study uses color-encoding prompts to map generated images to label masks, analyzes robustness via sampling variability and encoding shuffles, and explores performance on a harder dataset to illuminate limitations and future research directions. Overall, the work provides a novel, objective regime for assessing visual reasoning, generalization, and interpretability in multimodal generation systems, informing future benchmarks and multimodal curricula.

Abstract

Multi-modal large language models that have image output are emerging. Many image generation benchmarks focus on aesthetics instead of fine-grained generation capabilities. In PixelArena, we propose using semantic segmentation tasks to objectively examine their fine-grained generative intelligence with pixel precision. We find the latest Gemini 3 Pro Image has emergent image generation capabilities that generate semantic masks with high fidelity under zero-shot settings, showcasing visual intelligence unseen before and true generalization in new image generation tasks. We further investigate its results, compare them qualitatively and quantitatively with those of other models, and present failure cases. The findings not only signal exciting progress in the field but also provide insights into future research related to multimodality, reasoning, interpretability and benchmarking.

Paper Structure

This paper contains 18 sections, 1 equation, 17 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Segmentation and Face Parsing
  4. Image Generation Benchmarks and Metrics
  5. Segmentation with MLLMs
  6. Methods
  7. Dataset
  8. Models and Mask Generation
  9. Analysis
  10. Qualitative Comparisons
  11. Quantitative Results and Examining Data Contamination
  12. Further Failure Analysis and Pretended Reflections
  13. On a more challenging dataset
  14. Limitations and Discussion
  15. Conclusion
  16. ...and 3 more sections

Figures (17)

  • Figure 1: Palette of the standard color encodings from CelebAMask-HQ CelebAMask-HQ
  • Figure 2: Comparison between the Results of Different Models on celeb. Not cherrypicked.
  • Figure 3: Best prediction across celeb by geminipro with F1 score $0.7030$.
  • Figure 4: Worst prediction across celeb by geminipro with F1 score $0.0805$ and parallel attempts.
  • Figure 5: Comparison between the reference mask and masks predicted by two strong models.
  • ...and 12 more figures