From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration

TL;DR

Long-tail data imbalance in large vision-language models (LVLMs) hampers generalization, particularly on tail concepts. The authors introduce an Adaptive Data Refinement Framework (ADR) that first rebalances data by Data Rebalancing (DR) and then synthesizes tail data via Language Data Synthesis and Diffusion-Based Visual Data Synthesis (DS). DR uses a probability dictionary derived from four entity perspectives (Token, Object, Co-occurrence, Interrogation) and reverse indexing to filter redundant head data, while DS generates tail-representative data through synonyms, rewriting, and diffusion-driven image-caption-conversation pipelines. Across eleven benchmarks, ADR yields an average improvement of $4.36\%$ over strong baselines like LLaVA 1.5 without increasing data volume, with especially strong gains on tail concepts, demonstrating improved cross-modal understanding and generalization.

Abstract

Large Vision-Language Models (LVLMs) have achieved significant progress in combining visual comprehension with language generation. Despite this success, the training data of LVLMs still suffers from Long-Tail (LT) problems, where the data distribution is highly imbalanced. Previous works have mainly focused on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as recognition and classification. Nevertheless, the exploration of LVLM (e.g. LLaVA) and more general tasks (e.g. Visual Question Answering and Visual Reasoning) remains under-explored. In this paper, we first conduct an in-depth analysis of the LT issues in LVLMs and identify two core causes: the overrepresentation of head concepts and the underrepresentation of tail concepts. Based on the above observation, we propose an $\textbf{A}$daptive $\textbf{D}$ata $\textbf{R}$efinement Framework ($\textbf{ADR}$), which consists of two stages: $\textbf{D}$ata $\textbf{R}$ebalancing ($\textbf{DR}$) and $\textbf{D}$ata $\textbf{S}$ynthesis ($\textbf{DS}$). In the DR stage, we adaptively rebalance the redundant data based on entity distributions, while in the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and scarce images to supplement underrepresented portions. Through comprehensive evaluations across eleven benchmarks, our proposed ADR effectively mitigates the long-tail problem in the training data, improving the average performance of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.

Figures (15)

• Figure 1: Performance before and after addressing the LT problem. Our method surpasses the baseline over all benchmarks and also effectively improves the performance of tail 30% concepts.
• Figure 2: The overview of our Adaptive Data Refinement Framework (ADR). (a) In the Analyzing Stage, we first extract tokens, objects, co-occurrences, and interrogations from the training instances, then construct corresponding distribution using a reverse-indexed mapping. (b) In the Data Rebalancing stage, we analyze the optimizing direction and adaptively rebalance the redundant data based on the entity distribution identified in the Analyzing stage. (c) Finally, in the Data Synthesis stage, we utilize DDPM and the latent representations of scarce image instances to synthesize the underrepresented data.
• Figure 3: Long-tail distribution in instruction-tuning and benchmark datasets: (a) Token-level distribution in MME fu2023MME. (b) Token-level distribution in InstructMix665K liu2024visual. (c) Object-level distribution in MME fu2023MME. (d) Object-level distribution in InstructMix665K liu2024visual.
• Figure 4: Error accumulation curve of POPE and MME based on the training data distribution. It reveals that tail entities contribute to the majority of failure cases. (a) Token-level word distribution in MME fu2023MME and POPE Li2023POPE. (b) Object-level word distribution in MME and POPE. (c) Co-occurrence-level word distribution in MME and POPE.
• Figure 5: Ablation study on data rebalancing combinations. T, O, C, and W refer to Token, Object, Co-occurrence, and Interrogation respectively. The values displayed in the graph represent average scores across a variety of comprehensive benchmarks. The blue dashed line indicates the baseline performance of LLaVA 1.5.