Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Image

TL;DR

The paper addresses the challenge of generating coherent counterfactual T2I scenes with multiple concepts by introducing Replace in Translation (RIT), a latent-space, stepwise replacement framework guided by Explicit Logical Narrative Prompt (ELNP). A DeepSeek-driven ELNP generator and a question-block verification mechanism enable progressive, verifiable replacements to align all prompt concepts in complex scenes. To evaluate multi-entity counterfactuals, the authors extend the LC-Mis benchmark and introduce Multi-Concept Variance and Targeted Entities Coverage metrics, providing a specialized evaluation suite beyond pairwise metrics. Empirical results show state-of-the-art performance across 2–5 concept and mixed scenarios, with high entity-coverage and robust alignment, while acknowledging remaining challenges at very high concept counts. The work advances controllable T2I for imaginative, reliable counterfactual content and offers datasets, code, and templates to support further research.

Abstract

Text-to-Image (T2I) has been prevalent in recent years, with most common condition tasks having been optimized nicely. Besides, counterfactual Text-to-Image is obstructing us from a more versatile AIGC experience. For those scenes that are impossible to happen in real world and anti-physics, we should spare no efforts in increasing the factual feel, which means synthesizing images that people think very likely to be happening, and concept alignment, which means all the required objects should be in the same frame. In this paper, we focus on concept alignment. As controllable T2I models have achieved satisfactory performance for real applications, we utilize this technology to replace the objects in a synthesized image in latent space step-by-step to change the image from a common scene to a counterfactual scene to meet the prompt. We propose a strategy to instruct this replacing process, which is called as Explicit Logical Narrative Prompt (ELNP), by using the newly SoTA language model DeepSeek to generate the instructions. Furthermore, to evaluate models' performance in counterfactual T2I, we design a metric to calculate how many required concepts in the prompt can be covered averagely in the synthesized images. The extensive experiments and qualitative comparisons demonstrate that our strategy can boost the concept alignment in counterfactual T2I.

Figures (8)

• Figure 1: The qualitative examples of some counterfactual text prompts with more than two entities. Both SDXL sdxl and MoCE moce cannot cover every concepts while our method succeeds to fully cover all the concepts mentioned in the prompts.
• Figure 2: The overview of the ELNP and Question Blocks strategy for counterfactual T2I.
• Figure 3: An example of replacement from original base to the target counterfactual text prompt: A cat astronaut is riding a horse on the moon.
• Figure 4: The prompt template to generate ELNP: order of replacements. Based on the previous parsing toward entities in the exemplar, we can give clear instruction exemplars for DeepSeek to generate the ordered series instructing the controllable model to replace entities within the image step-by-step.
• Figure 5: A question block. $I$ is the intermediate image decoded from the latent space. Pass judge will check whether the positive answers cover no less than 60% of all questions in this block.