RealStats: A Rigorous Real-Only Statistical Framework for Fake Image Detection

TL;DR

RealStats addresses the challenge of detecting AI-generated images with interpretability and robustness to distribution shifts by proposing a real-only statistical framework. It frames fake image detection as a statistical hypothesis testing problem against the real-image distribution, using ECDF-based two-sided $p$-values for a diverse set of training-free statistics and aggregating them with Stouffer’s test or a minimum-$p$ ensemble. The method builds a null-distribution model by extracting statistics across detectors under perturbations, then selects an independent clique of statistics via pairwise tests and a KS uniformity check to ensure valid aggregation. Empirically, RealStats achieves competitive AUC and AP relative to training-free baselines, while providing interpretable outputs, scalability, and resilience to corruptions and modest domain shifts.

Abstract

As generative models continue to evolve, detecting AI-generated images remains a critical challenge. While effective detection methods exist, they often lack formal interpretability and may rely on implicit assumptions about fake content, potentially limiting robustness to distributional shifts. In this work, we introduce a rigorous, statistically grounded framework for fake image detection that focuses on producing a probability score interpretable with respect to the real-image population. Our method leverages the strengths of multiple existing detectors by combining training-free statistics. We compute p-values over a range of test statistics and aggregate them using classical statistical ensembling to assess alignment with the unified real-image distribution. This framework is generic, flexible, and training-free, making it well-suited for robust fake image detection across diverse and evolving settings.

Figures (14)

• Figure 1: Illustration of the score interpretability gap between a supervised classifier wang2020cnn and our statistical method. Top: A supervised model outputs scores that can separate real from fake images, but these scores are not inherently interpretable, as they lack clear statistical meaning and are often overconfident. Bottom: Our method produces calibrated $p$-values based on real image distributions. These values have a precise interpretation: the probability of observing such a result if the image were real. This enables principled decisions using a standard significance level.
• Figure 2: Illustration of the adaptability gap in existing training-free methods. Each row shows statistics from a different method (top: manifold curvature; bottom: permutation-based features), and each column corresponds to a different test condition. Top: The manifold curvature statistic shows opposite behavior across generators. In (a) SDXL, real images have higher curvature than real ones, while in (b) StyleGAN2, the pattern reverses. Bottom: The permutation-based feature statistic is sensitive to the perturbation strength $\lambda$. In (c), real images yield higher scores than fakes, but in (d), the ordering flips. These shifts highlight that handcrafted statistics often rely on assumptions, such as fixed magnitude differences, that do not generalize across models or test settings.
• Figure 3: Overview of the null distribution modeling phase. (a) Real images from the reference dataset $\mathcal{D}_{\text{real}}$ are processed using multiple detector configurations $(f_j, \lambda_k)$, producing scalar statistics whose empirical distributions are estimated and stored as ECDFs. (b) Pairwise statistical dependence among statistics is assessed via $\chi^2$ tests over real samples. (c) The resulting relationships define an independence graph, where edges indicate accepted independence. (d) A maximal clique is extracted and regularized via a uniformity constraint to select a subset $\mathcal{I} \subseteq \mathcal{S}$ of independent statistics.
• Figure 4: Overview of the inference phase. (a) A candidate image is processed using only the subset of detector configurations corresponding to the selected statistics $\mathcal{I}$. (b) Each resulting statistic is mapped to a two-sided $p$-value using the stored ECDFs. (c) The set of $p$-values is aggregated using a statistical aggregation method, such as Stouffer’s test. (d) The final decision is taken by comparing the unified $p$-value to a predefined significance level.
• Figure 5: Per-generator AUC comparison across methods shown in radar format.