Synthetic Perception: Can Generated Images Unlock Latent Visual Prior for Text-Centric Reasoning?

TL;DR

This work systematically investigates whether images generated on-the-fly by Text-to-Image (T2I) models can serve as a mechanism to unlock latent visual priors for text-centric reasoning, demonstrating its viability as a pathway to enrich language understanding in traditionally unimodal scenarios.

Abstract

A significant ``modality gap" exists between the abundance of text-only data and the increasing power of multimodal models. This work systematically investigates whether images generated on-the-fly by Text-to-Image (T2I) models can serve as a mechanism to unlock latent visual priors for text-centric reasoning. Through a comprehensive evaluation framework on text classification, we analyze the impact of critical variables, including T2I model quality (e.g., Flux.1, SDXL), prompt engineering strategies, and multimodal fusion architectures. Our findings demonstrate that this ``synthetic perception" can yield significant performance gains by effectively projecting text into a visual semantic space, even when augmenting strong large language model baselines like Llama-3 and Qwen-2.5. We show that this approach serves as a form of cross-modal probing, mitigating the sensory deprivation inherent in pure text training. However, the effectiveness is highly conditional, depending on the semantic alignment between text and the generated image, the task's visual groundability, and the generative fidelity of the T2I model. Our work establishes a rigorous benchmark for this paradigm, demonstrating its viability as a pathway to enrich language understanding in traditionally unimodal scenarios.

Figures (5)

• Figure 1: Overview of our evaluation framework for assessing the viability of using T2I-generated images as a complementary modality in text-centric learning tasks. The framework consists of three main stages: (1) Synthetic Visual Modality Generation, where images are generated on-demand from source text using various T2I models and prompt strategies; (2) Multimodal Representation and Fusion, where text and image features are extracted and combined through different fusion mechanisms; and (3) Downstream Task Evaluation, where the fused representations are evaluated on classification tasks against strong baselines.
• Figure 2: Qualitative comparison of different image generation strategies. The figure shows two cases: a concrete object description from Amazon Reviews (top row) and an abstract concept from AG News (bottom row). From left to right, we compare a baseline retrieved image (B4) with images generated by increasingly sophisticated T2I configurations. The visual quality and semantic alignment with the source text (indicated by the CLIP Score below each image) improve with better T2I models (SD1.5 vs. SDXL) and more advanced prompting strategies (P2 vs. P4), providing visual support for the quantitative results in Table \ref{['tab:t2i_prompt_impact']}.
• Figure 3: Normalized confusion matrices for B1 (left) and Ours (right) on the Amazon Reviews test set.
• Figure 4: Normalized confusion matrices for B1 (left) and Ours (right) on the AG News test set.
• Figure 5: Visualization of Cross-Attention Maps. The heatmaps show the attention weight distribution of specific text tokens (e.g., "red", "powerful") over the generated image regions. Darker red/blue indicates higher attention weight.