Unicorn: Text-Only Data Synthesis for Vision Language Model Training

TL;DR

Unicorn tackles the cost and scalability barrier of vision-language model training by proposing a text-only, cross-integrated three-stage data synthesis pipeline that yields Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for instruction-tuning. It leverages seed diversification, advanced LLM-based caption and instruction generation, and modality representation transfer to map text-derived representations into a visual space, enabling training of Unicorn-8B without real images. Experiments show Unicorn-8B attains competitive results on standard benchmarks, with substantial reductions in data generation cost and storage, evidencing the viability of text-only data for multimodal learning. Limitations include residual modality-gap noise and gaps in domain-specific knowledge, suggesting avenues for improving synthetic representations and knowledge integration.

Abstract

Training vision-language models (VLMs) typically requires large-scale, high-quality image-text pairs, but collecting or synthesizing such data is costly. In contrast, text data is abundant and inexpensive, prompting the question: can high-quality multimodal training data be synthesized purely from text? To tackle this, we propose a cross-integrated three-stage multimodal data synthesis framework, which generates two datasets: Unicorn-1.2M and Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we construct 1.2M semantically diverse high-quality captions by expanding sparse caption seeds using large language models (LLMs). In Stage 2: Instruction-Tuning Data Generation, we further process 471K captions into multi-turn instruction-tuning tasks to support complex reasoning. Finally, in Stage 3: Modality Representation Transfer, these textual captions representations are transformed into visual representations, resulting in diverse synthetic image representations. This three-stage process enables us to construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for instruction-tuning, without relying on real images. By eliminating the dependency on real images while maintaining data quality and diversity, our framework offers a cost-effective and scalable solution for VLMs training. Code is available at https://github.com/Yu-xm/Unicorn.git.

Figures (7)

• Figure 1: Unlike traditional image-text data synthesis frameworks, Unicorn removes the dependency on real image data, offering a more efficient and scalable solution by cutting down API costs, synthesis time, and storage requirements.
• Figure 2: Unicorn’s text-only data synthesis pipeline, comprising three cross-integrated stages, (1) Diverse Caption Data Synthesis, (2) Instruction-Tuning Data Synthesis, and (3) Modality Presentation Transfer, ultimately yields two synthetic datasets: Unicorn-1.2M and Unicorn-471K-Instruction. Notably, unlike traditional approaches that rely on real images, Unicorn generates image representations paired with text, entirely free of real image data.
• Figure 3: Data formats for the three instruction-tuning tasks. Each dialogue’s answer is derived from the textual content of diverse caption.
• Figure 4: Training and inference processes of Unicorn-8B. Training aligns synthetic image representations with LLM embeddings, while inference adjusts real image embeddings as input.
• Figure 5: Comparison of the data length distributions between Unicorn-1.2M and ShareGPT4V.