Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models
Yanbing Zeng, Jia Wang, Hanghang Ma, Junqiang Wu, Jie Zhu, Xiaoming Wei, Jie Hu
TL;DR
Forge-and-Quench tackles the challenge of leveraging multimodal understanding to improve image generation fidelity. It introduces a dual-conditioning framework: an enhanced text prompt $t^*$ and a forged Bridge Feature $e_b$ produced by a Bridge Adapter, which are injected into a frozen T2I backbone through an Injection Adapter. The approach preserves the MLLM's reasoning while enabling fine-grained visual guidance, validated by improvements in fidelity metrics (e.g., COCO-30K FID, GPT-Fidelity) and human judgments across multiple backbones. Ablation studies show diffusion-based Bridge Adapters and SigLIP-based Bridge Features yield the best balance of fidelity and robustness, with the method proving extensible and low-overhead. Overall, Forge-and-Quench offers a practical, scalable path to tighter integration of understanding and generation in unified multimodal models.
Abstract
Integrating image generation and understanding into a single framework has become a pivotal goal in the multimodal domain. However, how understanding can effectively assist generation has not been fully explored. Unlike previous works that focus on leveraging reasoning abilities and world knowledge from understanding models, this paper introduces a novel perspective: leveraging understanding to enhance the fidelity and detail richness of generated images. To this end, we propose Forge-and-Quench, a new unified framework that puts this principle into practice. In the generation process of our framework, an MLLM first reasons over the entire conversational context, including text instructions, to produce an enhanced text instruction. This refined instruction is then mapped to a virtual visual representation, termed the Bridge Feature, via a novel Bridge Adapter. This feature acts as a crucial link, forging insights from the understanding model to quench and refine the generation process. It is subsequently injected into the T2I backbone as a visual guidance signal, alongside the enhanced text instruction that replaces the original input. To validate this paradigm, we conduct comprehensive studies on the design of the Bridge Feature and Bridge Adapter. Our framework demonstrates exceptional extensibility and flexibility, enabling efficient migration across different MLLM and T2I models with significant savings in training overhead, all without compromising the MLLM's inherent multimodal understanding capabilities. Experiments show that Forge-and-Quench significantly improves image fidelity and detail across multiple models, while also maintaining instruction-following accuracy and enhancing world knowledge application. Models and codes are available at https://github.com/YanbingZeng/Forge-and-Quench.