Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Prompt Optimization for Dataset-Level Feature Discovery

Adrian Cosma, Oleg Szehr, David Kletz, Alessandro Antonucci, Olivier Pelletier

TL;DR

The paper reframes feature discovery from unstructured text as a dataset-level prompt optimization problem, introducing a multi-agent framework that jointly proposes, extracts, and evaluates a global set of interpretable features to optimize downstream classification. By grounding feedback at the dataset level rather than per example, and by incorporating interpretability signals through a ReflectiveProposer, the approach yields more robust and grounded features than per-sample methods or hand-crafted prompts. Empirical results across financial sentiment and toxicity datasets show consistent improvements in F1 and feature interpretability, while preventing leakage of the target label into features. The work demonstrates the practicality and benefits of dataset-level prompt optimization for automatic, interpretable feature engineering, with limitations including computational cost and dependence on instruction-following quality of the underlying models.

Abstract

Feature extraction from unstructured text is a critical step in many downstream classification pipelines, yet current approaches largely rely on hand-crafted prompts or fixed feature schemas. We formulate feature discovery as a dataset-level prompt optimization problem: given a labelled text corpus, the goal is to induce a global set of interpretable and discriminative feature definitions whose realizations optimize a downstream supervised learning objective. To this end, we propose a multi-agent prompt optimization framework in which language-model agents jointly propose feature definitions, extract feature values, and evaluate feature quality using dataset-level performance and interpretability feedback. Instruction prompts are iteratively refined based on this structured feedback, enabling optimization over prompts that induce shared feature sets rather than per-example predictions. This formulation departs from prior prompt optimization methods that rely on per-sample supervision and provides a principled mechanism for automatic feature discovery from unstructured text.

Automatic Prompt Optimization for Dataset-Level Feature Discovery

TL;DR

The paper reframes feature discovery from unstructured text as a dataset-level prompt optimization problem, introducing a multi-agent framework that jointly proposes, extracts, and evaluates a global set of interpretable features to optimize downstream classification. By grounding feedback at the dataset level rather than per example, and by incorporating interpretability signals through a ReflectiveProposer, the approach yields more robust and grounded features than per-sample methods or hand-crafted prompts. Empirical results across financial sentiment and toxicity datasets show consistent improvements in F1 and feature interpretability, while preventing leakage of the target label into features. The work demonstrates the practicality and benefits of dataset-level prompt optimization for automatic, interpretable feature engineering, with limitations including computational cost and dependence on instruction-following quality of the underlying models.

Abstract

Feature extraction from unstructured text is a critical step in many downstream classification pipelines, yet current approaches largely rely on hand-crafted prompts or fixed feature schemas. We formulate feature discovery as a dataset-level prompt optimization problem: given a labelled text corpus, the goal is to induce a global set of interpretable and discriminative feature definitions whose realizations optimize a downstream supervised learning objective. To this end, we propose a multi-agent prompt optimization framework in which language-model agents jointly propose feature definitions, extract feature values, and evaluate feature quality using dataset-level performance and interpretability feedback. Instruction prompts are iteratively refined based on this structured feedback, enabling optimization over prompts that induce shared feature sets rather than per-example predictions. This formulation departs from prior prompt optimization methods that rely on per-sample supervision and provides a principled mechanism for automatic feature discovery from unstructured text.
Paper Structure (22 sections, 2 equations, 5 figures, 1 table, 2 algorithms)

This paper contains 22 sections, 2 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Annotation using LMs
  4. Feature Discovery using LMs
  5. Prompt Optimization
  6. Method
  7. Problem Definition
  8. The Optimization Process
  9. Modules Used for Prompt Optimization
  10. Overview
  11. Asymptotic Analysis of the Language-based Program
  12. Experimental Setup
  13. Backbone SLMs.
  14. Datasets.
  15. Implementation Details.
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Diagram of proposed LM agents for instructions optimization. (1) The FeatureProposer module outputs a set of feature definitions, based on the current prompt and a set of examples. (2) Based on provided definitions, an Extractor is instantiated which extracts text features across the entire dataset and are evaluated. (3) Textual feedback is gathered by InterpretabilityScorer and PerformanceFeedback modules. (4) Based on the feedback, new instructions for the FeatureProposer are proposed by the ReflectiveProposer.
  • Figure 2: Comparison between ReflectiveProposer and variants using scalar feedback, across Yahoo and Twitter News, for four pretrained SLM models. Optimizing for F1 scores only can lead to feature leakage, artificially increasing performance, while regularizing through interpretability feedback can result in more grounded features.
  • Figure 3: Evaluation on ToxicChat dataset, considering the input-only toxicity subset.
  • Figure 4: Evolution of accuracy by increasing number of examples sets, and, in turn, of prompt proposals, for Qwen-4B and Qwen-14B, across Yahoo News and Twitter News datasets.
  • Figure 5: Performance evolution with increasing the number of iterative prompt refinements using the ReflectiveProposer on the Yahoo News dataset.