Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Models
Yi Wang, Mushui Liu, Wanggui He, Hanyang Yuan, Longxiang Zhang, Ziwei Huang, Guanghao Zhang, Wenkai Fang, Haoze Jiang, Shengxuming Zhang, Dong She, Jinlong Liu, Weilong Dai, Mingli Song, Hao Jiang, Jie Song
TL;DR
The paper tackles the difficulty of generating complex, compositional images with unified multimodal models by introducing a Functionality-oriented Expert (FoXperts) architecture and a Multimodal Chain of Thought (MCoT) framework. FoXperts decouple understanding and generation across text and vision, mitigating function-domain conflicts, while MCoT decomposes image creation into Planning, Acting, Reflection, and Correction, trained via a multi-task, disentangled paradigm. Empirical results on GenEval, T2I-CompBench, MS-COCO, and image-understanding benchmarks show FoX with MCoT achieving state-of-the-art or strong competitiveness with only 1.3B parameters, along with robust ablations confirming the contribution of each component. The work advances complex image synthesis by combining explicit, modular reasoning with function-aware specialization, improving both generation fidelity and alignment in multimodal tasks.
Abstract
Unified generative models have shown remarkable performance in text and image generation. For image synthesis tasks, they adopt straightforward text-to-image (T2I) generation. However, direct T2I generation limits the models in handling complex compositional instructions, which frequently occur in real-world scenarios. Although this issue is vital, existing works mainly focus on improving the basic image generation capability of the models. While such improvements help to some extent, they still fail to adequately resolve the problem. Inspired by Chain of Thought (CoT) solving complex problems step by step, this work aims to introduce CoT into unified generative models to address the challenges of complex image generation that direct T2I generation cannot effectively solve, thereby endowing models with enhanced image generation ability. To achieve this, we first propose Functionality-oriented eXperts (FoXperts), an expert-parallel architecture in our model FoX, which assigns experts by function. FoXperts disentangles potential conflicts in mainstream modality-oriented designs and provides a solid foundation for CoT. When introducing CoT, the first question is how to design it for complex image generation. To this end, we emulate a human-like artistic workflow--planning, acting, reflection, and correction--and propose the Multimodal Chain of Thought (MCoT) approach, as the data involves both text and image. To address the subsequent challenge of designing an effective MCoT training paradigm, we develop a multi-task joint training scheme that equips the model with all capabilities required for each MCoT step in a disentangled manner. This paradigm avoids the difficulty of collecting consistent multi-step data tuples. Extensive experiments show that FoX consistently outperforms existing unified models on various T2I benchmarks, delivering notable improvements in complex image generation.