Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment

TL;DR

This work tackles hallucinations and misalignment in large vision-language models by introducing SynthAlign, a synthetic-data pipeline for post-training alignment guided by Direct Preference Optimization (DPO). The method generates synthetic images via Stable Diffusion, ranks them with a reward signal, crafts instruction prompts with GPT-4o, produces responses from open-source LVLMs, and selects top and bottom responses using a reward function to train with DPO on data sampled from a small synthetic corpus. By replacing costly human annotation with reward-model proxies, SynthAlign delivers substantial improvements on vision-language and hallucination benchmarks (e.g., MMHal-Bench and POPE) with a 9K–12K data budget. The approach demonstrates scalable, safer LVLM deployment potential while highlighting the importance of data quality and the trade-offs inherent in synthetic data for real-world applications.

Abstract

Large Vision-Language Models (LVLMs) have shown promising capabilities in understanding and generating information by integrating both visual and textual data. However, current models are still prone to hallucinations, which degrade the performance and greatly harm the user experience in real-world applications. Post-training alignment, particularly preference-tuning, is intended to align model outputs and behaviors (safety, instruction-following, style), ensuring robustness and adaptability to a wide range of tasks. The use of synthetic data for alignment, particularly in multimodal settings, remains under explored. Existing approaches typically use a strong model or a ground-truth model (CLIP) to determine positive and negative image-text data points. This paper proposes SynthAlign, a pipeline to generate and collect synthetic human-preference image-text data with optimal control built specifically for post-training alignment with DPO. At the core of the framework is the utilization of reward models as a proxy of human preference. A series of evaluation and benchmarking is provided to validate the effectiveness of the proposed framework and the resulting dataset. Notably, our framework enhanced LLaVA-1.5-7B achieved substantial POPE improvements: 87.6\% accuracy and 97.8\% precision, MMHal-Bench score increased from 2.36 to 3.49, and hallucination rate decreased from 51.0\% to 25.0\% (a 50.98\% relative reduction).

Figures (5)

• Figure 1: Overview of the framework: Starting with an initial text-to-image prompt, the Stable Diffusion model generates synthetic images. These images are then filtered using a reward model to exclude low-quality samples and retain only those with the highest scores. The selected images, along with their corresponding instruction prompts, serve as input for open-source LVLMs to generate responses. These responses are evaluated based on various criteria, and only the highest-scoring ones are selected to identify the most suitable positive and negative pairs for DPO-based training.
• Figure 2: The figure illustrates image generation results using Stable Diffusion across four different guidance scales (5.0, 7.0, 9.0, 11.0), with the highest-scored image selected for each prompt based on a preference model evaluation.
• Figure 3: Sample of preferred and dispreferred responses. The dispreferred ones include speculative, redundant, or unclear information. This highlights the quality distinctions in vision-language task performance in the context of preference learning.
• Figure 4: Qualitative demonstrations comparing our method with GPT4-V.
• Figure 5: Selection percentage of preferred guidance scales by the reward model.