Think Bright, Diffuse Nice: Enhancing T2I-ICL via Inductive-Bias Hint Instruction and Query Contrastive Decoding

TL;DR

This work tackles two core bottlenecks in Text-to-Image In-Context Learning (T2I-ICL): compliance failure, where models ignore the query in favor of superficial context, and prior-dominated hallucination, where outputs reflect priors rather than the given context. It introduces TBDN, a training-free framework that jointly employs Hint Instruction (HI) to inject a context-aware inductive bias and Query Contrastive Decoding (QCD) to suppress priors during decoding, forming a closed-loop that aligns rule understanding with query-driven generation. Through extensive experiments on CoBSAT, Text-to-Image Fast Mini-ImageNet, and Dreambench++, TBDN demonstrates state-of-the-art or competitive performance with robust generalization across LVLM backbones and prompts, while avoiding costly training. The results suggest a practical, scalable pathway for reliable T2I-ICL by leveraging prompt-based biasing and decoding-time constraints in a diffusion-based generation pipeline. Limitations include reliance on textual guidance rather than end-to-end multimodal training and potential gaps in fine-grained visual detail alignment, pointing to future work in extending HI and QCD to broader MLLM setups and tasks.

Abstract

Text-to-Image In-Context Learning (T2I-ICL) enables customized image synthesis via interleaved text-image examples but faces two mutually reinforcing bottlenecks, compliance failure and prior-dominated hallucination, that form a vicious cycle degrading generation quality. Existing methods rely on tailored training, which limits flexibility and raises deployment costs. To address these challenges effectively, we propose TBDN, a training-free framework integrating two complementary closed-loop mechanisms: Hint Instruction (HI) and Query Contrastive Decoding (QCD). HI injects task-aware inductive bias via lightweight prompt engineering to anchor models on contextual mapping rules, thereby mitigating compliance failure. QCD adjusts the decoding distributions of language models by contrasting full-input and query-omitted distributions, suppressing prior-dominated hallucination. TBDN achieves State-of-the-Art performance on CoBSAT and Text-to-Image Fast Mini-ImageNet, with robust generalization across model backbones, prompt designs, and hyperparameters. It also maintains promising performance in concept preservation and prompt following on Dreambench++. By breaking the two bottlenecks, TBDN establishes a simple yet effective framework for efficient and reliable T2I-ICL.

Figures (15)

• Figure 1: Overview of the TBDN framework for T2I-ICL. Given an instruction (enclosed in '[ ]'), context examples, and queries (enclosed in '( )'), TBDN first injects task-aware inductive bias via Hint Instruction, then refines outputs via Query Contrastive Decoding to suppress priors, and finally generates images with a diffusion model.
• Figure 2: Two critical bottlenecks in T2I-ICL (evaluated on CoBSAT). Compliance failure (left): methods parrot input context (e.g., "hat", "cup") instead of reasoning query semantics. Prior-dominated hallucination (right) methods generate prior-aligned outputs (e.g., "red/green apples") that violate input requirements.
• Figure 3: Compliance failure error counts across methods on CoBSAT.
• Figure 4: Overview of Hint Instruction, a mechanism guiding LVLMs to prioritize context-aware query reasoning, thereby mitigating compliance failure.
• Figure 5: An illustrative example of prior-dominated hallucination in LVLMs for T2I-ICL. Given text-image pairs and the final query, the LVLM exhibits reliance on its prior association (apple $\leftrightarrow$ red) to generate an incorrect description, leading the diffusion model to render a red apple that mismatches the semantics of the ground truth answer. This failure motivates our Query Contrastive Decoding (QCD) method, which mitigates such hallucination by adjusting LVM output distributions.