Iterative Feature Space Optimization through Incremental Adaptive Evaluation

TL;DR

This work tackles the bottlenecks of iterative feature-space optimization by introducing EASE, a generalized adaptive feature space evaluator that decouples information via a Feature-Sample Subspace Generator and captures complex interactions with a Contextual Attention Evaluator. EASE enables incremental updates, leveraging overlapping information between consecutive feature spaces to accelerate optimization while preserving prior evaluation knowledge. Through extensive experiments on 14 real-world datasets, EASE demonstrates superior feature-space refinement, improved predictive performance, and notable efficiency gains across multiple iterative frameworks. The framework offers robust generalization, adaptability, and practical impact for scalable feature-space engineering in diverse domains.

Abstract

Iterative feature space optimization involves systematically evaluating and adjusting the feature space to improve downstream task performance. However, existing works suffer from three key limitations:1) overlooking differences among data samples leads to evaluation bias; 2) tailoring feature spaces to specific machine learning models results in overfitting and poor generalization; 3) requiring the evaluator to be retrained from scratch during each optimization iteration significantly reduces the overall efficiency of the optimization process. To bridge these gaps, we propose a gEneralized Adaptive feature Space Evaluator (EASE) to efficiently produce optimal and generalized feature spaces. This framework consists of two key components: Feature-Sample Subspace Generator and Contextual Attention Evaluator. The first component aims to decouple the information distribution within the feature space to mitigate evaluation bias. To achieve this, we first identify features most relevant to prediction tasks and samples most challenging for evaluation based on feedback from the subsequent evaluator. This decoupling strategy makes the evaluator consistently target the most challenging aspects of the feature space. The second component intends to incrementally capture evolving patterns of the feature space for efficient evaluation. We propose a weighted-sharing multi-head attention mechanism to encode key characteristics of the feature space into an embedding vector for evaluation. Moreover, the evaluator is updated incrementally, retaining prior evaluation knowledge while incorporating new insights, as consecutive feature spaces during the optimization process share partial information. Extensive experiments on fourteen real-world datasets demonstrate the effectiveness of the proposed framework. Our code and data are publicly available.

Figures (9)

• Figure 1: (a) Illustration of the iterative feature space optimization, where the optimization module refines the feature space based on the feedback of the evaluator until the optimal one is identified. (b) The feature spaces between consecutive iterations exhibit informational overlap.
• Figure 2: Framework overview and parameter update for EASE. The framework comprises two key components: the Feature-Sample Subspace Generator and the Contextual Attention Evaluator. The first component aims to decouple the complex information within the feature space, enabling the evaluator to focus on capturing the most challenging aspects for evaluation. The second component is designed to comprehensively capture the characteristics of the feature space, ensuring fair and accurate evaluation. By considering information overlap across consecutive iterations, the evaluator incrementally updates its parameters, enhancing the efficiency of the overall optimization process.
• Figure 3: Time complexity comparison of different feature space evaluators across various datasets.
• Figure 4: Comparison of prediction performance between original and EASE refined feature spaces.
• Figure 5: Comparison of refinement performance of feature space evaluators within FLSR.