Open Multimodal Retrieval-Augmented Factual Image Generation

TL;DR

ORIG addresses factual inconsistency in image generation by introducing an open multimodal retrieval-augmented framework for Factual Image Generation (FIG). It formalizes FIG, builds a three-module pipeline for open web-based retrieval, prompt construction, and generation, and introduces FIG-Eval to benchmark factual grounding across perceptual, compositional, and temporal dimensions. Experimental results show that ORIG consistently improves factual consistency and image quality over strong baselines by effectively integrating textual and visual evidence. This approach demonstrates the practical potential of open multimodal retrieval for knowledge-grounded image synthesis in dynamic real-world contexts.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable progress in generating photorealistic and prompt-aligned images, but they often produce outputs that contradict verifiable knowledge, especially when prompts involve fine-grained attributes or time-sensitive events. Conventional retrieval-augmented approaches attempt to address this issue by introducing external information, yet they are fundamentally incapable of grounding generation in accurate and evolving knowledge due to their reliance on static sources and shallow evidence integration. To bridge this gap, we introduce ORIG, an agentic open multimodal retrieval-augmented framework for Factual Image Generation (FIG), a new task that requires both visual realism and factual grounding. ORIG iteratively retrieves and filters multimodal evidence from the web and incrementally integrates the refined knowledge into enriched prompts to guide generation. To support systematic evaluation, we build FIG-Eval, a benchmark spanning ten categories across perceptual, compositional, and temporal dimensions. Experiments demonstrate that ORIG substantially improves factual consistency and overall image quality over strong baselines, highlighting the potential of open multimodal retrieval for factual image generation.

Figures (7)

• Figure 1: Motivation of factual image generation (FIG) with open multimodal retrieval. (a) Reliance on internal knowledge alone often leads to outdated or hallucinated content. (b) Incorporating external information improves grounding but remains constrained by static and unimodal sources. (c) Leveraging open retrieval of multimodal evidence integrates evolving knowledge and complementary cues to achieve FIG.
• Figure 2: The overall pipeline of the ORIG framework. ORIG adaptively controls multimodal retrieval and prompt construction, dynamically deciding whether to continue retrieval or proceed based on the current state of accumulated knowledge.
• Figure 3: Question-answering scores for different modality retrieval tasks (evaluated by GPT-5), with red indicating incorrect evaluations.
• Figure 4: The generation results for prompt "Generate a picture of a frog's life cycle" with three retrieval methods
• Figure 5: Different granularity for prompt "Generate a picture of a Qin Dynasty emperor riding in a horse-drawn carriage."