UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation

TL;DR

UtilGen tackles the problem of data augmentation for downstream tasks by introducing a utility-centric paradigm that uses downstream feedback to guide generation. It presents Task-Oriented Data Valuation to estimate per-sample utility via a meta-learned weight network, and a dual-level optimization consisting of Model-Level Generation Capability Optimization and Instance-Level Generation Policy Optimization to adapt both the generator and generation policies. Across eight benchmarks, UtilGen yields a average accuracy improvement of 3.87% over prior SOTA when training on synthetic data alone and maintains gains in joint training settings, demonstrating more impactful and task-relevant synthetic data. The work highlights a practical shift toward task utility in data augmentation, with broad implications for efficient data synthesis and deployment across diverse architectures and tasks.

Abstract

Data augmentation using generative models has emerged as a powerful paradigm for enhancing performance in computer vision tasks. However, most existing augmentation approaches primarily focus on optimizing intrinsic data attributes -- such as fidelity and diversity -- to generate visually high-quality synthetic data, while often neglecting task-specific requirements. Yet, it is essential for data generators to account for the needs of downstream tasks, as training data requirements can vary significantly across different tasks and network architectures. To address these limitations, we propose UtilGen, a novel utility-centric data augmentation framework that adaptively optimizes the data generation process to produce task-specific, high-utility training data via downstream task feedback. Specifically, we first introduce a weight allocation network to evaluate the task-specific utility of each synthetic sample. Guided by these evaluations, UtilGen iteratively refines the data generation process using a dual-level optimization strategy to maximize the synthetic data utility: (1) model-level optimization tailors the generative model to the downstream task, and (2) instance-level optimization adjusts generation policies -- such as prompt embeddings and initial noise -- at each generation round. Extensive experiments on eight benchmark datasets of varying complexity and granularity demonstrate that UtilGen consistently achieves superior performance, with an average accuracy improvement of 3.87% over previous SOTA. Further analysis of data influence and distribution reveals that UtilGen produces more impactful and task-relevant synthetic data, validating the effectiveness of the paradigm shift from visual characteristics-centric to task utility-centric data augmentation.

Figures (8)

• Figure 1: Comparison of high-utility samples within the same category (Persian cats) across two different tasks. White Persian cats (left) are more useful in Task 1, while golden ones (right) are more beneficial in Task 2, highlighting the diverse data requirements in different downstream tasks.
• Figure 2: The UtilGen framework for feedback-driven data augmentation, comprising three key stages: (1) Task-Oriented Data Valuation (Sec. \ref{['sec:todv']}); (2) Model-Level Generation Capability Optimization (Sec. \ref{['sec:mlpo']}); (3) Instance-Level Generation Policy Optimization (Sec. \ref{['sec:ilo']}).
• Figure 3: (a) Feature space visualization on the Flower dataset nilsback2008automatedshows that our synthetic data achieves closer alignment with the real data distribution compared to vanilla Stable Diffusion. (b) Utility-aware weight distributions for synthetic and real data on the Flower dataset nilsback2008automated, showing sample utility scores for downstream tasks.
• Figure 4: Comparison of synthetic images generated by SD v2.1, GIF zhang2023expanding, GAP yeo2024controlled, DataDream kim2024datadream, and UtilGen.
• Figure 5: Synthetic Data Scaling Effects across different training data regimes. (a) Models trained exclusively on synthetic data (Synthetic Data Only). (b) Models trained on combined synthetic and real data (Synthetic + Real Data).