Constantly Improving Image Models Need Constantly Improving Benchmarks

TL;DR

ECHO reframes benchmark construction by extracting real-world image-generation prompts and feedback from social media, creating a responsive, reusable benchmark that evolves with model releases. By applying ECHO to GPT-4o Image Gen, the authors assemble a dataset of over $31{,}000$ prompts that reveal novel tasks, better separate state-of-the-art models from alternatives, and yield community-driven metrics (e.g., color shift, identity preservation, structure) to guide future improvements. The framework combines broad post collection, context reconstruction, and multimodal processing (including VLMs) to produce self-contained samples with quality labels, while annotating failures and interaction patterns to enrich evaluation beyond templated tasks. Automatic evaluation via a VLM-based judge, supplemented by limited human validation, demonstrates meaningful model differentiation and shows how user feedback can close the loop for metric design and model development. Overall, ECHO offers a reproducible, crowd-sourced approach to keeping benchmarks aligned with evolving user needs and capabilities of modern image-generation systems.

Abstract

Recent advances in image generation, often driven by proprietary systems like GPT-4o Image Gen, regularly introduce new capabilities that reshape how users interact with these models. Existing benchmarks often lag behind and fail to capture these emerging use cases, leaving a gap between community perceptions of progress and formal evaluation. To address this, we present ECHO, a framework for constructing benchmarks directly from real-world evidence of model use: social media posts that showcase novel prompts and qualitative user judgments. Applying this framework to GPT-4o Image Gen, we construct a dataset of over 31,000 prompts curated from such posts. Our analysis shows that ECHO (1) discovers creative and complex tasks absent from existing benchmarks, such as re-rendering product labels across languages or generating receipts with specified totals, (2) more clearly distinguishes state-of-the-art models from alternatives, and (3) surfaces community feedback that we use to inform the design of metrics for model quality (e.g., measuring observed shifts in color, identity, and structure). Our website is at https://echo-bench.github.io.

Figures (24)

• Figure 1: ECHO distills collective discussion about a new generative model into a structured benchmark. As a case study, we apply ECHO to GPT-4o Image Gen openai2025 on Twitter/X. Left: ECHO automatically surfaces highly diverse and novel tasks not covered in prior benchmarks. Right: Consequently, our image-to-image split shows a 3.2x larger relative performance gap compared to a prior image editing benchmark, GEdit liu2025step1x.
• Figure 2: ECHO Framework. ECHO is motivated by several challenges inherent to social media. (1) We start with broad queries followed by relevance filtering, since basic querying presents a volume-relevance tradeoff. (2) We then extract prompts from these posts, making sure to utilize the full post tree, as context can be spread across posts. (3) We then apply multimodal processing, since useful data also exists in non-standard formats. (4) Finally, we reserve the highest quality data for benchmarking, while the rest is used for analysis.
• Figure 3: Dataset Comparison. The prompts in ECHO are significantly different from prior benchmarks. Top: the image-to-image split is more diverse and complex, with more unique first bigrams. Bottom: the text-to-image split is more fluent, as measured by perplexity under Pythia 12B biderman2023pythia.
• Figure 4: Common Failures Observed by Users. A word cloud of failure cases, derived from community feedback, showing practical capabilities that users care about in real use cases and curiosity-driven tests that reveal deeper model limitations. Common failures include identity shift, color drift, text rendering errors, and style mismatches; more exploratory failures include originality and reasoning about volume.
• Figure 5: How Users Interact with Models. We depict qualitative examples of (a) user solutions and (b) exploratory behaviors, discovered via community feedback. We also visualize (c) activity trends using the timestamps of collected posts.