From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

TL;DR

The paper tackles the cost of optimizing data augmentation by proposing a performance-guided, closed-loop system that grounds LLM-driven transform synthesis in empirical downstream accuracy. By building a labeled repository of over $6{,}000$ augmentations and fine-tuning with LoRA, the approach achieves up to $600\times$ fewer evaluated candidates compared to brute-force search while maintaining competitive peak accuracy. Key findings show that the model internalizes semantic performance cues rather than memorizing syntax, and that direct prompting outperforms Chain-of-Thought prompting in this task. This framework offers a scalable, data-driven path toward autonomous, task-aligned data preprocessing, with potential extensions to other architectures and multimodal settings.

Abstract

Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches. We introduce a performance-aware, closed-loop solution in the NNGPT ecosystem of projects that enables LLMs to autonomously engineer optimal transformations by internalizing empirical performance cues. We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions, each annotated solely by downstream model accuracy. Training uses pairwise performance ordering (better-worse transformations), enabling alignment through empirical feedback without reinforcement learning, reward models, or symbolic objectives. This reduces the need for exhaustive search, achieving up to 600x times fewer evaluated candidates than brute-force discovery while maintaining competitive peak accuracy and shifting generation from random synthesis to task-aligned design. Ablation studies show that structured Chain-of-Thought prompting introduces syntactic noise and degrades performance, whereas direct prompting ensures stable optimization in performance-critical code tasks. Qualitative and quantitative analyses demonstrate that the model internalizes semantic performance cues rather than memorizing syntax. These results show that LLMs can exhibit task-level reasoning through non-textual feedback loops, bypassing explicit symbolic rewards.

Figures (8)

• Figure 1: Brute-force data transformation generation and evaluation pipeline for constructing an LLM fine-tuning dataset. Image transformation functions are automatically generated, evaluated under a fixed training configuration, and stored with their corresponding accuracy, yielding a performance-labeled dataset used in subsequent LLM fine-tuning.
• Figure 2: The iterative performance-guided fine-tuning loop. In each cycle, the LLM synthesizes candidate data transformations which are empirically validated via downstream model training. High-fidelity metadata from these trials is used to update model parameters via LoRA, inducing a semantic alignment between generative output and empirical performance cues.
• Figure 3: Accuracy distribution of data transformation functions grouped by the number of selected transforms. Single-transform configurations exhibit higher mean accuracy and lower variance compared to compositions of multiple transforms.
• Figure 4: Mean and maximum accuracy of generated transformations across fine-tuning epochs. The mean accuracy exhibits a steadily upward trajectory (r = 0.34) over successive epochs, indicating improved overall generation quality, while the maximum accuracy remains relatively stable, suggesting consistent rediscovery of high-performing transformations.
• Figure 5: Evolution of Generated Transformation Quality. Kernel Density Estimation (KDE) of validation accuracy across fine-tuning stages. The shift from Early (blue) to Late (red) epochs indicates the model effectively minimizes the generation of low-performing code and converges on a high-performance semantic region