PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensembles

TL;DR

PolyGen tackles the synthetic data gap in vision-language pre-training by enforcing generator-invariance through a polylithic ensemble of diverse diffusion models and a structured hard-negative curriculum. It combines Structured Caption Pairs, a Multi-Generator Synthesis Ensemble, and Curriculum-Based Training with TripletCLIP losses to promote semantic alignment across heterogeneous visual manifolds and improve compositional reasoning. Empirically, it achieves up to +19.0% Delta_MTL over SynthCLIP and notable gains on SugarCrepe-based benchmarks, illustrating that manifold diversity can be a more data-efficient scaling strategy than sheer volume. The work underscores the value of controlled synthetic data, safety-conscious generation, and curriculum-driven learning for robust, open-world vision-language understanding with practical societal benefits.

Abstract

Synthetic data offers a scalable solution for vision-language pre-training, yet current state-of-the-art methods typically rely on scaling up a single generative backbone, which introduces generator-specific spectral biases and limits feature diversity. In this work, we introduce PolyGen, a framework that redefines synthetic data construction by prioritizing manifold coverage and compositional rigor over simple dataset size. PolyGen employs a Polylithic approach to train on the intersection of architecturally distinct generators, effectively marginalizing out model-specific artifacts. Additionally, we introduce a Programmatic Hard Negative curriculum that enforces fine-grained syntactic understanding. By structurally reallocating the same data budget from unique captions to multi-source variations, PolyGen achieves a more robust feature space, outperforming the leading single-source baseline (SynthCLIP) by +19.0% on aggregate multi-task benchmarks and on the SugarCrepe++ compositionality benchmark (+9.1%). These results demonstrate that structural diversity is a more data-efficient scaling law than simply increasing the volume of single-source samples.

Figures (15)

• Figure 1: PolyGen Pipeline Overview.Stage 1 (Polylithic Synthesis): Concepts from MetaCLIP Bank and semantic axes feed into a dual-LLM system. Mistral V0.2 generates base captions $T^+$ conditioned on concept-attribute pairs, while Llama 3.1 produces hard negatives $T^-$ by modifying specific semantic dimensions. Caption pairs are rendered through four diffusion models: Diversity Experts (SD 1.5, SD 2) ensure broad manifold coverage despite lower fidelity, while Recognizability Experts (SANA, SDXL-Turbo) provide high prompt adherence despite limited variance. Stage 2 (Curriculum-Based Training): A scheduler linearly increases hard negative proportion $p$ from 0 to 0.5 during training. Unused samples populate a leftover queue, maintaining data efficiency. Batches combine multi-positive samples ($I^+, T^+$) and hard negatives ($I^-, T^-$) to compute $\mathcal{L}_{I2I}$ and $\mathcal{L}_{HN}$.
• Figure 2: Conceptual Illustration of the Generator-Invariance Hypothesis. Individual generators (colored circles) act as "Experts" with specific biases: older models like SD 1.5 (Blue) offer high diversity but low fidelity, while modern distilled models like SDXLT (Red) offer high fidelity but suffer from mode collapse. PolyGen (Green Dashed) trains on the union of these manifolds, approximating the coverage of the Real Data Distribution (Grey) by marginalizing out generator-specific artifacts.
• Figure 3: Hard Negative Generation Across Multi-Generator Ensemble. Two examples of controlled semantic perturbations rendered through four diffusion models. Top: Concept-level modification ("oreo" $\rightarrow$ "chocolate chip cookie", $a =$ Concept). Bottom: Viewpoint transformation ("bird's eye view" $\rightarrow$ "sidewalk view", $a =$ Perspective). Diversity Experts (SD 1.5, SD 2) produce high intra-concept variance with lower photorealism, while Recognizability Experts (SANA, SDXL-Turbo) generate photorealistic outputs with reduced stylistic variation. The architectural heterogeneity forces models to learn generator-invariant semantic representations, as the only consistent signal across columns is the underlying concept modification.
• Figure 4: Recognizability vs. Diversity, with color indicating downstream performance. While individual models trade off diversity or recognizability, PolyGen ensembles break this Pareto frontier, maximizing both axes to achieve superior $\Delta_{MTL}$.
• Figure 5: Zero-shot programmatic conditioning. (a) Baseline prompt enforcing structured concept-attribute tuples $(c, a)$ to ensure semantic orthogonality. (b) Hard Negative prompt imposing strict counterfactual constraints to generate valid $T^-$ samples without syntactic drift.