TIP and Polish: Text-Image-Prototype Guided Multi-Modal Generation via Commonality-Discrepancy Modeling and Refinement

TL;DR

This work tackles thematic coherence and style consistency in multi-modal generation by explicitly modeling semantic commonality and discrepancy between text, images, and their prototypes. The TIPPo framework introduces a text-image-prototype signal pipeline, a Dual Alignment Attention mechanism, and a Difference Operator to enrich and refine visual information. It augments this with a prototype-driven unsupervised contrastive loss and PolishPPO, delivering hierarchical literal- and semantic-level supervision during fine-tuning. Empirical results on ArtMuse and COCO-CN show improved automatic metrics and favorable LLM-based evaluations, validating the approach and its training strategies.

Abstract

Multi-modal generation struggles to ensure thematic coherence and style consistency. Semantically, existing methods suffer from cross-modal mismatch and lack explicit modeling of commonality and discrepancy. Methods that rely on fine-grained training fail to balance semantic precision with writing style consistency. These shortcomings lead to suboptimal generation quality. To tackle these issues, we propose \textbf{\textit{TIPPo}}, a simple yet effective framework with explicit input modeling and comprehensive optimization objectives. It extracts the input text and images via multi-modal encoder and adapters, then measures the visual prototype. \textbf{T}extual, \textbf{I}mage, and \textbf{P}rototype signals are then fed to our proposed Dual Alignment Attention and Difference Operator modules before language model decoding. The proposed \textbf{Po}lishPPO reinforces the style consistency, while the unsupervised contrastive learning during SFT mitigates inter-sample representation collapse. Experimental results demonstrate the promising performance of \textbf{\textit{TIPPo}} in automatic evaluation and LLM-based criteria for creativity and semantic consistency.

Figures (3)

• Figure 1: The overview of TIPPo framework
• Figure 2: Effectiveness of individual modules or methods in TIPPo evaluated by automatic metric on ArtMUSE.
• Figure 3: LLM evaluation of different methods on ArtMUSE.