Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

TL;DR

This work introduces Mimic Score, a weight-space geometry-based data-quality metric that aligns a training sample's negative gradient with a reference model's direction toward an optimal weight configuration. Built on Mimic Score, Grad-Mimic provides a two-stage framework: Stage 1 online re-weighting during training and Stage 2 offline aggregation to curate high-value data, achieving improved data efficiency and faster convergence. Empirical results across six image datasets and DataComp-scale CLIP pretraining show strong performance gains, robust mislabeled-sample detection, and substantial data-filtering benefits with minimal overhead. The approach leverages publicly available pretrained weights, avoids validation-set dependencies, and offers practical scalability with broad applicability to data curation and model training pipelines.

Abstract

Large-scale web-crawled datasets contain noise, bias, and irrelevant information, necessitating data selection techniques. Existing methods depend on hand-crafted heuristics, downstream datasets, or require expensive influence-based computations -- all of which limit scalability and introduce unwanted data dependencies. To address this, we introduce the Mimic Score, a simple and geometry-based data-quality metric that evaluates utility by measuring alignment between a sample's gradients and a target direction induced by a pre-trained reference model. This leverages readily available model weights, avoids needing validation datasets, and incurs minimal computational overheads. Building on this metric, we propose Grad-Mimic, a two-stage framework that re-weights samples online to accelerate training and aggregates sample utilities offline to construct effective data filters. Empirically, we show that using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7% fewer training steps. Additionally, mimic score-based filters augment existing filtering techniques, enabling improved CLIP models trained with 4.7 million fewer samples.

Figures (10)

• Figure 1: High-value vs. low-value samples identified by Mimic Score: Random samples from the top 5% (first row) and bottom 5% (second row) of web-crawled data, ranked by their mimic score. Each image includes its caption and CLIP score (higher indicates better quality). Mimic scores align closely with CLIP scores. High-value samples generally have detailed captions and coherent visuals, while low-value ones carry short captions and misaligned content.
• Figure 2: Grad-Mimic Two-stage Workflow: During training, Grad-Mimic measures the alignment between each sample’s negative gradient and an induced vector from pre-trained reference model, then uses normalized alignment to re-weight gradients. After training, these alignment signals, mimic scores, are aggregated to identify low-value samples and used to construct an ensemble filter.
• Figure 3: Grad-Mimic outperforms influence function-based methods: Leveraging the geometric information of the reference model as a selection guide provides several advantages, including higher effectiveness, lower computational cost, and greater accessibility.
• Figure 4: Mimic Score Distribution: Distributions extracted from CIFAR100 clearly separate by label correctness.
• Figure 5: The left figure shows mimic score-guided training converges faster than vanilla baseline. The middle one uses retention rate to estimate training dataset quality under mislabeled sample experiments. The right figure takes the average on mimic scores in each curated dataset for performance gain estimation. [S/M] denotes the scale of DataComp dataset, small or medium, that the filter is designed on.

Theorems & Definitions (8)

• Lemma 9.1
• proof
• Lemma 9.2
• proof
• Lemma 9.3
• proof
• Theorem 9.4
• proof