ASemConsist: Adaptive Semantic Feature Control for Training-Free Identity-Consistent Generation

TL;DR

A novel framework, ASemconsist, that addresses this challenge through selective text embedding modification, enabling explicit semantic control over character identity without sacrificing prompt alignment, and proposes a unified evaluation protocol, the Consistency Quality Score (CQS), which integrates identity preservation and per-image text alignment into a single comprehensive metric.

Abstract

Recent text-to-image diffusion models have significantly improved visual quality and text alignment. However, generating a sequence of images while preserving consistent character identity across diverse scene descriptions remains a challenging task. Existing methods often struggle with a trade-off between maintaining identity consistency and ensuring per-image prompt alignment. In this paper, we introduce a novel framework, ASemconsist, that addresses this challenge through selective text embedding modification, enabling explicit semantic control over character identity without sacrificing prompt alignment. Furthermore, based on our analysis of padding embeddings in FLUX, we propose a semantic control strategy that repurposes padding embeddings as semantic containers. Additionally, we introduce an adaptive feature-sharing strategy that automatically evaluates textual ambiguity and applies constraints only to the ambiguous identity prompt. Finally, we propose a unified evaluation protocol, the Consistency Quality Score (CQS), which integrates identity preservation and per-image text alignment into a single comprehensive metric, explicitly capturing performance imbalances between the two metrics. Our framework achieves state-of-the-art performance, effectively overcoming prior trade-offs. Project page: https://minjung-s.github.io/asemconsist

Figures (13)

• Figure 1: Overview of the consistent story generation task and our results. The identity prompt, provides the shared character description across all images. The per-image descriptions, specify unique attributes for each image. Our results show consistent character representation across images, with each per-image description effectively reflected.
• Figure 2: Overall overview of our method. (a) illustrates the problem setting and text embedding modification, while (b) shows our adaptive feature sharing strategy that leverages image features to resolve ambiguity.
• Figure 3: Cosine similarity distributions of residual features from prompt sets $\{p_{\mathrm{id}}, p_1\}, \dots, \{p_{\mathrm{id}}, p_k\}$ at different transformer blocks. At ambiguity-distinctive blocks (yellow box), high-ambiguity (red; samples with high DreamSim scores) and low-ambiguity sets (green; samples with low DreamSim scores) are clearly separable. At non-distinctive blocks, distributions overlap, hindering ambiguity discrimination.
• Figure 4: Analysis and validation of our text embedding modification. We use about 100 samples from the benchmark for this analysis. In the generated sample figure, the identity prompts are shown in pink, and per-image prompts are shown in blue.
• Figure 5: Qualitative comparison. The identity prompt (pink text) describes the character identity shared across the image sequence, while per-image prompts (green) specify attributes unique to each image.