Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming

TL;DR

The paper tackles unsupervised feature transformation (EUFT) in domains where labels are costly, proposing a duet-play framework of generator-critic LLM agents that leverage in-context learning to derive pseudo-supervision from unlabeled data. The critic diagnoses the dataset to provide textual guidance, the generator produces tokenized feature transformations guided by this advice, and iterative refinement between the agents enhances the feature space without labeled data, with potential human expert collaboration. Empirical results across 12 public datasets show the method often surpasses supervised baselines in transformation efficiency, robustness, and applicability, while maintaining favorable time performance. The framework is characterized by interpretability through token sequences and extendability to conversational or human-in-the-loop settings, offering a practical pathway to robust, scalable EUFT in data-limited environments.

Abstract

Feature transformation involves generating a new set of features from the original dataset to enhance the data's utility. In certain domains like material performance screening, dimensionality is large and collecting labels is expensive and lengthy. It highly necessitates transforming feature spaces efficiently and without supervision to enhance data readiness and AI utility. However, existing methods fall short in efficient navigation of a vast space of feature combinations, and are mostly designed for supervised settings. To fill this gap, our unique perspective is to leverage a generator-critic duet-play teaming framework using LLM agents and in-context learning to derive pseudo-supervision from unsupervised data. The framework consists of three interconnected steps: (1) Critic agent diagnoses data to generate actionable advice, (2) Generator agent produces tokenized feature transformations guided by the critic's advice, and (3) Iterative refinement ensures continuous improvement through feedback between agents. The generator-critic framework can be generalized to human-agent collaborative generation, by replacing the critic agent with human experts. Extensive experiments demonstrate that the proposed framework outperforms even supervised baselines in feature transformation efficiency, robustness, and practical applicability across diverse datasets.

Figures (8)

• Figure 1: Framework overview. We implement feature generation through a duet-play generator-critic framework. We also extend it to a conversational generation manner.
• Figure 2: We provide a response example of critic agent on the dataset playground. We obtain advice from semantic and data perspectives.
• Figure 3: We present a response example of generator agent on the dataset playground. The feature sequence represent a dataset and the generated features interpret the semantic meanings.
• Figure 4: Ablation study. (a) We study the impact of using different LLMs in LPFG. (b) We investigate the performance of generator guided by different information.
• Figure 5: Robustness check. On diabetes, we investigate the robustness of the proposed method when different downstream ML models are employed.