HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models

TL;DR

HYPE-EDIT-1 addresses a key gap in evaluating frontier image-editing models by measuring reliability and total practical cost under realistic retry scenarios. It introduces a 100-task benchmark with 10 independent outputs per task, split into 50 public and 50 private tasks, and uses both human judges and a visual-language model as evaluative signals. The framework defines $P@1$, $P@10$, expected attempts under a retry cap, and a cost metric that combines model generation and human review, enabling comparisons of high-cost, high-reliability models against cheaper ones with more retries. The benchmark provides open tooling, a standard JSON schema, and a public/private split to support reproducible, longitudinal tracking of reliability improvements in marketing/design edits, with results showing substantial variability across models and nontrivial effective costs driven by retries.

Abstract

Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation, plus a standardized JSON schema and tooling for VLM and human-based judging. Across the evaluated models, per-attempt pass rates span 34-83 percent and effective cost per success spans USD 0.66-1.42. Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews.

Figures (5)

• Figure 1: Qualitative comparisons between Gemini 3 Pro Preview (Nano Banana Pro) and Seedream 4.5. Each panel shows the input reference(s) alongside 10-sample collages for each model. The collage view makes it easy to see how repeated attempts at the same task can diverge, even for narrowly scoped edits such as remove or swap where minimal variation is expected. In practice, these collages often show partial instruction compliance or structure drift that would be hidden by selecting a single best output.
• Figure 2: Example tasks from HYPE-EDIT-1 showing diverse editing instructions across different task types: change, remove, restructure, and enhance. Each panel shows the input reference image(s) with the instruction and task category.
• Figure 3: Combined split summary charts (human majority labels; VLM judge used only as a check).
• Figure 4: Public split summary charts (human majority labels).
• Figure 5: Private split summary charts (human majority labels).