Multimodal Benchmarking and Recommendation of Text-to-Image Generation Models

TL;DR

This work introduces a reproducible, open-source framework for benchmarking text-to-image generation under uniform conditions, focusing on metadata-augmented prompts to improve garment realism and semantic fidelity on the DeepFashion-MultiModal dataset. It integrates a comprehensive set of metrics (CLIP-based alignment, LPIPS, FID, retrieval) with qualitative analyses and demonstrates that metadata-enriched prompts enhance realism and grounding across diverse model families (diffusion, transformer-based, and hybrids). The study provides a large-scale cross-model comparison (10+ models) and shows that top performers achieve robust trade-offs between realism and semantic accuracy, while offering an interactive Streamlit-based recommendation tool. The findings support metadata-driven prompt design as a practical strategy to boost T2I quality in complex, fashion-driven domains, and lay the groundwork for future standardized evaluation, model fine-tuning, and user-personalized generation pipelines.

Abstract

This work presents an open-source unified benchmarking and evaluation framework for text-to-image generation models, with a particular focus on the impact of metadata augmented prompts. Leveraging the DeepFashion-MultiModal dataset, we assess generated outputs through a comprehensive set of quantitative metrics, including Weighted Score, CLIP (Contrastive Language Image Pre-training)-based similarity, LPIPS (Learned Perceptual Image Patch Similarity), FID (Frechet Inception Distance), and retrieval-based measures, as well as qualitative analysis. Our results demonstrate that structured metadata enrichments greatly enhance visual realism, semantic fidelity, and model robustness across diverse text-to-image architectures. While not a traditional recommender system, our framework enables task-specific recommendations for model selection and prompt design based on evaluation metrics.

Figures (16)

• Figure 1: Visual comparison of generated images across models for Prompt 1 (Base vs Metadata)
• Figure 2: Visual comparison of generated images across models for Prompt 2 (Base vs Metadata)
• Figure 3: Architecture diagrams for Latent Diffusion Models: LDM and SDXL
• Figure 4: Multi-stage diffusion pipeline in CogView3 ding2022cogview3 using text expansion and relay upscaling
• Figure 5: Architectures of PixArt-$\alpha$chen2023pixart and GDT wang2024gdt within the Gated/Group Diffusion cluster. PixArt uses transformer blocks with AdaLN; GDT leverages shared attention over grouped prompts.