Feature Selection as Deep Sequential Generative Learning

TL;DR

The paper tackles the generalization gap in feature selection by reframing the problem as sequential generative learning, where a feature subset is encoded into a continuous embedding and decoded autoregressively. The VTFS framework integrates a variational transformer encoder, a decoder, and an evaluator to learn a feature-subset embedding space, a gradient-guided search to locate high-utility embeddings, and an autoregressive generator to produce the best subset; it also leverages a reinforcement-learning-based data collector to bootstrap diverse training experiences. Key contributions include a multi-loss objective that couples reconstruction, evaluation, and regularization, a gradient-steered optimization mechanism, and extensive experiments on 16 real-world datasets showing consistent improvements and robustness across models. This approach offers a scalable, model-agnostic alternative to discrete search, with potential broad applicability across domains and tasks where feature selection is critical.

Abstract

Feature selection aims to identify the most pattern-discriminative feature subset. In prior literature, filter (e.g., backward elimination) and embedded (e.g., Lasso) methods have hyperparameters (e.g., top-K, score thresholding) and tie to specific models, thus, hard to generalize; wrapper methods search a feature subset in a huge discrete space and is computationally costly. To transform the way of feature selection, we regard a selected feature subset as a selection decision token sequence and reformulate feature selection as a deep sequential generative learning task that distills feature knowledge and generates decision sequences. Our method includes three steps: (1) We develop a deep variational transformer model over a joint of sequential reconstruction, variational, and performance evaluator losses. Our model can distill feature selection knowledge and learn a continuous embedding space to map feature selection decision sequences into embedding vectors associated with utility scores. (2) We leverage the trained feature subset utility evaluator as a gradient provider to guide the identification of the optimal feature subset embedding;(3) We decode the optimal feature subset embedding to autoregressively generate the best feature selection decision sequence with autostop. Extensive experimental results show this generative perspective is effective and generic, without large discrete search space and expert-specific hyperparameters.

Figures (9)

• Figure 1: Our perspective can be viewed as a sequence generation (b) rather than as an iterative discrete selection (a).
• Figure 2: An overview of VTFS. First, we employ the variational transformer-based sequential model to construct feature subset embedding space. Second, we search for better embeddings by moving local optimal embeddings along the gradient direction maximizing the downstream predictive accuracy. Third, we generate the feature token sequences in an autoregressive manner based on these better embeddings and keep the best one with the highest downstream ML performance.
• Figure 3: Reinforcement data collector.
• Figure 4: Analysis of the impact of different feature subset embedding modules on feature selection.
• Figure 5: Analysis of the impact of data collector on selecting the effective feature subset.