Beyond the Pixels: VLM-based Evaluation of Identity Preservation in Reference-Guided Synthesis

TL;DR

The paper tackles the challenge of quantitatively evaluating identity preservation in reference-guided synthesis, where traditional global embeddings fail to capture fine-grained identity cues between $I_1$ and $I_2$. It introduces CHARIS, a hierarchical, transformation-based evaluation framework that grounds VLM reasoning in verifiable visual evidence via a decomposition: type/style → attributes → features, aided by an External Knowledge Base (EKB). The authors validate CHARIS across four state-of-the-art generation systems and demonstrate stronger alignment with human judgments than embedding-based metrics, supported by a comprehensive benchmark of 1,078 prompts spanning 154 subjects with 5–6 concurrent transformations per prompt. This approach yields interpretable diagnostics of identity drift and enables robust evaluation across underrepresented subjects and artistic styles, with practical implications for improving identity preservation in real-world, multi-axes creative workflows.

Abstract

Evaluating identity preservation in generative models remains a critical yet unresolved challenge. Existing metrics rely on global embeddings or coarse VLM prompting, failing to capture fine-grained identity changes and providing limited diagnostic insight. We introduce Beyond the Pixels, a hierarchical evaluation framework that decomposes identity assessment into feature-level transformations. Our approach guides VLMs through structured reasoning by (1) hierarchically decomposing subjects into (type, style) -> attribute -> feature decision tree, and (2) prompting for concrete transformations rather than abstract similarity scores. This decomposition grounds VLM analysis in verifiable visual evidence, reducing hallucinations and improving consistency. We validate our framework across four state-of-the-art generative models, demonstrating strong alignment with human judgments in measuring identity consistency. Additionally, we introduce a new benchmark specifically designed to stress-test generative models. It comprises 1,078 image-prompt pairs spanning diverse subject types, including underrepresented categories such as anthropomorphic and animated characters, and captures an average of six to seven transformation axes per prompt.

Figures (10)

• Figure 1: Comparison of different generative models (DSD, Omnigen, Anystory, UNO) illustrating gaps between automated metrics and human judgment in character identity.
• Figure 2: Prompt for extracting Type
• Figure 3: Prompt for extracting Style
• Figure 4: Given a specific style and type we consult EKB to create a prompt to extract visible attributes.
• Figure 5: Given visible attributes we consult EKB to create a prompt to extract visible features.