Explainable AI-Generated Image Detection RewardBench

TL;DR

The paper addresses the need for explainable AI-generated image detection by evaluating Multimodal Large Language Models as reward models that judge explanations. It introduces XAIGID-RewardBench, a first domain-specific benchmark with approximately 3,000 human-annotated triplets (image plus two detector explanations) to assess how well judges rank explanations. Experimental results show a clear gap between current MLLM judges and human performance, with the best reward model reaching 88.76% 4-way accuracy versus 98.30% human agreement on ground-truth preferences, and reveal common failure modes such as artifact generalization and unconnected reasoning. These findings provide a foundation for improving explainable detection systems and highlight future directions toward more reliable, tool-enabled, and human-aligned AI forensics in images.

Abstract

Conventional, classification-based AI-generated image detection methods cannot explain why an image is considered real or AI-generated in a way a human expert would, which reduces the trustworthiness and persuasiveness of these detection tools for real-world applications. Leveraging Multimodal Large Language Models (MLLMs) has recently become a trending solution to this issue. Further, to evaluate the quality of generated explanations, a common approach is to adopt an "MLLM as a judge" methodology to evaluate explanations generated by other MLLMs. However, how well those MLLMs perform when judging explanations for AI-generated image detection generated by themselves or other MLLMs has not been well studied. We therefore propose \textbf{XAIGID-RewardBench}, the first benchmark designed to evaluate the ability of current MLLMs to judge the quality of explanations about whether an image is real or AI-generated. The benchmark consists of approximately 3,000 annotated triplets sourced from various image generation models and MLLMs as policy models (detectors) to assess the capabilities of current MLLMs as reward models (judges). Our results show that the current best reward model scored 88.76\% on this benchmark (while human inter-annotator agreement reaches 98.30\%), demonstrating that a visible gap remains between the reasoning abilities of today's MLLMs and human-level performance. In addition, we provide an analysis of common pitfalls that these models frequently encounter. Code and benchmark are available at https://github.com/RewardBench/XAIGID-RewardBench.

Figures (7)

• Figure 1: Overview of the XAIGID-RewardBench. With the capabilities of today's MLLMs, an AI-generated image detection response can include both a verdict and an explanation, providing much more insight into the reasoning process. Furthermore, MLLMs are also now being used to judge the quality of these explanations, but little work has been done to assess the quality of the MLLM judges themselves. Therefore, we present a human-annotated benchmark to evaluate how well these reward models (i.e., MLLM judges) score the explanations provided by MLLMs for AI-generated image detection.
• Figure 2: A visual depiction of the input (a triplet) for MLLM judges. Both responses come from SOTA MLLMs as AI-generated image detectors, yet they disagree with each other. An MLLM judge must deduce which one is more convincing given all 3 pieces of information (One image and two detection responses from detectors). MLLM judges currently still cannot achieve human performance on the stated task.
• Figure 3: Distribution of annotator choices. While Response 1 was picked 2.1% more often, the difference is not significant. This indicates that there was no significant position bias in the human annotation.
• Figure 4: Some failure cases of MLLM policy models.
• Figure 5: A conceptual illustration of our annotation tool. The image and response sources are hidden to remove bias. Annotators are also encouraged to look past the initial one-word verdict and focus more on the explanation given by MLLM detectors.