EXaMCaP: Subset Selection with Entropy Gain Maximization for Probing Capability Gains of Large Chart Understanding Training Sets

TL;DR

This work tackles the cost of assessing capability gains from ChartU datasets by proposing EXaMCaP, a three-stage subset selection framework that preserves knowledge diversity through entropy gain maximization. It combines Extreme Sample Filtering, clustering with K-Means, and entropy-based greedy sampling to build a high-entropy subset whose evaluation proxies full-set fine-tuning gains. Entropy is quantified via a Von Neumann entropy on a density matrix derived from Gaussian embeddings, with ρ = M / Tr(M) and $E(\rho) = -\mathrm{Tr}(\rho \ln \rho)$. Empirically, EXaMCaP achieves near-full performance using about 20% of the data across multiple ChartU benchmarks and MLLM architectures, enabling faster dataset iteration and informing data-generation directions for chart understanding research.

Abstract

Recent works focus on synthesizing Chart Understanding (ChartU) training sets to inject advanced chart knowledge into Multimodal Large Language Models (MLLMs), where the sufficiency of the knowledge is typically verified by quantifying capability gains via the fine-tune-then-evaluate paradigm. However, full-set fine-tuning MLLMs to assess such gains incurs significant time costs, hindering the iterative refinement cycles of the ChartU dataset. Reviewing the ChartU dataset synthesis and data selection domains, we find that subsets can potentially probe the MLLMs' capability gains from full-set fine-tuning. Given that data diversity is vital for boosting MLLMs' performance and entropy reflects this feature, we propose EXaMCaP, which uses entropy gain maximization to select a subset. To obtain a high-diversity subset, EXaMCaP chooses the maximum-entropy subset from the large ChartU dataset. As enumerating all possible subsets is impractical, EXaMCaP iteratively selects samples to maximize the gain in set entropy relative to the current set, approximating the maximum-entropy subset of the full dataset. Experiments show that EXaMCaP outperforms baselines in probing the capability gains of the ChartU training set, along with its strong effectiveness across diverse subset sizes and compatibility with various MLLM architectures.

Figures (7)

• Figure 1: (a) General training set generation pipeline and dataset capability gain probing with full-set finetuning, (b) performance of subsets is comparable to the full sets.
• Figure 2: The architecture of EXaMCaP. For the given ChartU training set, EXaMCaP first conducts preliminary extreme sample filtering based on PPL. Secondly, it performs entropy gain maximization sampling within each cluster from prior partitioning, and then obtains a subset with high entropy to maintain ChartU knowledge diversity. Finally, it performs subset-based capability gains probing.
• Figure 3: The impact of subset sizes on capability gain probing. The subset sizes range from 20K to 80K with an interval of 20K, and 60K acts as the dividing line for subsets that account for approximately 20% or less of the total training set size.
• Figure 4: The impact of alternative extraction and target MLLMs on capability gains probing, involving using different MLLMs obtain data information (such as embeddings), then using (a) LLaVA-Next-LLaMA3 and (b) Qwen2.5-VL for fine-tuning.
• Figure 5: Comparison of time costs under different configurations. The time cost is measured in hours on a computing node with 4 × RTX 4090 GPUs.