LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuning

TL;DR

This work tackles automatic data processing for domain-specific LLM fine-tuning under privacy constraints. It introduces LLM-AutoDP, which uses LLMs as agents to generate, evaluate, and iteratively refine data processing strategies, guided by group-relative feedback. To accelerate evaluation, it proposes Distribution-Preserving Sampling (DPS), Processing Target Selection (PTS), and Cache-and-Reuse Mechanism (CRM), achieving substantial speedups while maintaining pipeline quality. Experiments on medical datasets show that models trained on DP-processed data outperform unprocessed baselines by large margins (over 80% win rates) and beat AutoML baselines by about 65%, with up to 10x reductions in search time, highlighting both practical impact and privacy-preserving capabilities.

Abstract

Large Language Models (LLMs) can be fine-tuned on domain-specific data to enhance their performance in specialized fields. However, such data often contains numerous low-quality samples, necessitating effective data processing (DP). In practice, DP strategies are typically developed through iterative manual analysis and trial-and-error adjustment. These processes inevitably incur high labor costs and may lead to privacy issues in high-privacy domains like healthcare due to direct human access to sensitive data. Thus, achieving automated data processing without exposing the raw data has become a critical challenge. To address this challenge, we propose LLM-AutoDP, a novel framework that leverages LLMs as agents to automatically generate and optimize data processing strategies. Our method generates multiple candidate strategies and iteratively refines them using feedback signals and comparative evaluations. This iterative in-context learning mechanism enables the agent to converge toward high-quality processing pipelines without requiring direct human intervention or access to the underlying data. To further accelerate strategy search, we introduce three key techniques: Distribution Preserving Sampling, which reduces data volume while maintaining distributional integrity; Processing Target Selection, which uses a binary classifier to identify low-quality samples for focused processing; Cache-and-Reuse Mechanism}, which minimizes redundant computations by reusing prior processing results. Results show that models trained on data processed by our framework achieve over 80% win rates against models trained on unprocessed data. Compared to AutoML baselines based on LLM agents, LLM-AutoDP achieves approximately a 65% win rate. Moreover, our acceleration techniques reduce the total searching time by up to 10 times, demonstrating both effectiveness and efficiency.

Figures (5)

• Figure 1: The overall framework of LLM-AutoDP. The left part utilizes an LLM as an agent to iteratively refine the prompt, enabling the agent to generate high-quality data processing strategies. The right part consists of an evaluation module for generating strategy feedback scores and an acceleration module for speeding up data processing. The LLM iteratively controls the interaction between the two parts to select high-quality processing strategies.
• Figure 2: Visualization of the distribution of the original data and the data sampled by 20% on Huatuo‑26M‑Lite dataset. It can be seen that the distribution of the sampled data is consistent with that of the original data.
• Figure 3: Results of using Qwen3-32B and DeepSeek-R1-Distill-Llama-70B as the agent models. We repeat the experiments on each agent 5 times and record the win rate of the resulting fine-tuned models versus the models fine-tuned on the original, unprocessed data. The colored columns and black bars indicate the average and standard deviation of repeated experiment results. The top row and bottom row show results of using GPT-4 and Baichuan-M1-14B-Instruct as the judges, respectively.
• Figure 4: Results of using different numbers of strategies in the initial round. We use Qwen3-32B as the agent model and report the win rate of the resulting fine-tuned models versus the models fine-tuned on the original, unprocessed data. The top row and bottom row show results of using GPT-4 and Baichuan-M1-14B-Instruct as the judges, respectively.
• Figure 5: A step-by-step example of strategy optimization using LLM-AutoDP. A total of 5 rounds of iterations are performed, but only the details of 2 rounds are shown due to space imitation.