Routing to the Right Expertise: A Trustworthy Judge for Instruction-based Image Editing

TL;DR

Experimental results demonstrate that JURE is reliable by achieving superior alignment with human judgments, setting a new standard for automated IIE evaluation.

Abstract

Instruction-based Image Editing (IIE) models have made significantly improvement due to the progress of multimodal large language models (MLLMs) and diffusion models, which can understand and reason about complex editing instructions. In addition to advancing current IIE models, accurately evaluating their output has become increasingly critical and challenging. Current IIE evaluation methods and their evaluation procedures often fall short of aligning with human judgment and often lack explainability. To address these limitations, we propose JUdgement through Routing of Expertise (JURE). Each expert in JURE is a pre-selected model assumed to be equipped with an atomic expertise that can provide useful feedback to judge output, and the router dynamically routes the evaluation task of a given instruction and its output to appropriate experts, aggregating their feedback into a final judge. JURE is trustworthy in two aspects. First, it can effortlessly provide explanations about its judge by examining the routed experts and their feedback. Second, experimental results demonstrate that JURE is reliable by achieving superior alignment with human judgments, setting a new standard for automated IIE evaluation. Moreover, JURE's flexible design is future-proof - modular experts can be seamlessly replaced or expanded to accommodate advancements in IIE, maintaining consistently high evaluation quality. Our evaluation data and results are available at https://github.com/Cyyyyyrus/JURE.git.

Figures (3)

• Figure 1: Conceptual architecture of the JURE framework. JURE takes the original image, editing instruction "Add a black cat behind the boy", edited images generated by different IIE models (IP2P, HQ Edit, EMU Edit) as the initial input. In this example, JURE iteratively reasons about to verify (i) object presence (whether a cat was added), (ii) attribute accuracy (if added, whether it is black), (iii) spatial correctness (whether it appears behind the boy), and (iv) visual integrity (ensuring no unwanted edits or artifacts appear elsewhere). During each iteration (highlighted by different colors), the Orchestrator routes to the right expert, incrementally stores their output to the Context Dictionary for future reference, and dynamically decides its action for the next iteration. For instance, after detecting that IP2P failed to retain the boy in the first iteration, subsequent spatial analysis involving the boy and the cat is performed only for the HQ and EMU edits. Finally, it aggregates all expert responses to produce a final judgment.
• Figure 2: Inter-Evaluator Agreement (Cohen’s Kappa) Heatmap. Values represent agreement levels between evaluators, where higher values indicate stronger alignment.
• Figure 3: Expert Invocation Frequency in JURE-o1.