Eliminating Hallucination in Diffusion-Augmented Interactive Text-to-Image Retrieval

TL;DR

This work tackles diffusion-induced hallucinations in Diffusion-Augmented Interactive Text-to-Image Retrieval (DAI-TIR) by learning an encoder that acts as a semantic filter. It introduces Diffusion-aware Multi-view Contrastive Learning (DMCL), which constructs three views per query (text, diffusion proxy, and fused) and optimizes two complementary objectives: a multi-view query–target alignment with a generalized symmetric InfoNCE loss plus hard-negative mining, and a text–diffusion consistency objective to reduce cross-view drift. The approach yields a shared embedding space where intent-relevant cues are reinforced across views while diffusion-specific signals are suppressed, improving retrieval across five benchmarks and under distribution shift, with gains up to 7.37% in Hits@10 and a diffusion-augmented training dataset released for research. The results demonstrate that DMCL provides robust semantic alignment and cross-view consistency, enabling diffusion augmentation to enhance rather than derail interactive text-to-image retrieval. The work offers practical impact for more reliable, multi-round visual search and contributes a scalable dataset to support future diffusion-aware retrieval research.

Abstract

Diffusion-Augmented Interactive Text-to-Image Retrieval (DAI-TIR) is a promising paradigm that improves retrieval performance by generating query images via diffusion models and using them as additional ``views'' of the user's intent. However, these generative views can be incorrect because diffusion generation may introduce hallucinated visual cues that conflict with the original query text. Indeed, we empirically demonstrate that these hallucinated cues can substantially degrade DAI-TIR performance. To address this, we propose Diffusion-aware Multi-view Contrastive Learning (DMCL), a hallucination-robust training framework that casts DAI-TIR as joint optimization over representations of query intent and the target image. DMCL introduces semantic-consistency and diffusion-aware contrastive objectives to align textual and diffusion-generated query views while suppressing hallucinated query signals. This yields an encoder that acts as a semantic filter, effectively mapping hallucinated cues into a null space, improving robustness to spurious cues and better representing the user's intent. Attention visualization and geometric embedding-space analyses corroborate this filtering behavior. Across five standard benchmarks, DMCL delivers consistent improvements in multi-round Hits@10, reaching as high as 7.37\% over prior fine-tuned and zero-shot baselines, which indicates it is a general and robust training framework for DAI-TIR.

Figures (7)

• Figure 1: Semantic mismatch between diffusion-generated query images and the textual query due to hallucination.
• Figure 2: Text consistency and hallucination-type distribution of diffusion-generated images, evaluated by Qwen3-VL and Gemma3.
• Figure 3: DMCL Training Framework Overview
• Figure 4: The overall interactive retrieval performance (measured by cumulative Hits@10) across five benchmarks. Our proposed DMCL consistently outperforms baselines across all dialogue rounds.
• Figure 5: Qualitative Evidence for Conflict Detail Filtering: DAR vs. DMCL(Ours)