A Contrast Based Feature Selection Algorithm for High-dimensional Data set in Machine Learning

TL;DR

This paper tackles the challenge of feature selection in high-dimensional data by proposing ContrastFS, a model-free filter method that constructs dimensionless surrogate representations of each class using low-order moments and then scores features by the discrepancies between classes. By evaluating features individually with a greedy top-$m$ approach and addressing redundancy via correlation of discrepancy vectors, ContrastFS achieves fast computation and competitive accuracy across diverse datasets. The method demonstrates stability, scalability, and applicability as a preprocessing step, with empirical evidence showing substantial speed advantages over baselines and strong performance on both image-like and tabular data. The work contributes a practical, theory-backed criterion for discriminative feature selection and offers insights into using surrogate representations to study feature correlations and redundancy.

Abstract

Feature selection is an important process in machine learning and knowledge discovery. By selecting the most informative features and eliminating irrelevant ones, the performance of learning algorithms can be improved and the extraction of meaningful patterns and insights from data can be facilitated. However, most existing feature selection methods, when applied to large datasets, encountered the bottleneck of high computation costs. To address this problem, we propose a novel filter feature selection method, ContrastFS, which selects discriminative features based on the discrepancies features shown between different classes. We introduce a dimensionless quantity as a surrogate representation to summarize the distributional individuality of certain classes, based on this quantity we evaluate features and study the correlation among them. We validate effectiveness and efficiency of our approach on several widely studied benchmark datasets, results show that the new method performs favorably with negligible computation in comparison with other state-of-the-art feature selection methods.

Figures (7)

• Figure 1: Distinguish images by difference. We select 5 most important features from the MNIST dataset, these five pixels are located in the center of the images. We can see that these pixels show significant differences between the digits 0 and 8, both in the individual samples ((a) and (b)) and in the class averages ((c) and (d)).
• Figure 2: Representation of class 8 of the MNIST dataset. (a) shows the average of the samples belonging to digit 8. (b) shows the $l_2$ barycenter of the samples. (c) shows the normalized plot, which removes the mean and scales by the standard deviation. (d) shows the surrogate representation, which captures the deviation from the commonality of the original data by a dimensionless quantity.
• Figure 3: Logarithmic unit time cost vs. accuracy.
• Figure 4: Classification accuracy vs. number of selected features. This figure shows the average accuracy achieved by XGBoost classifieds fed in features selected by various methods in multiple runs on benchmark datasets.
• Figure 5: Accuracy of the feature subset after redundancy reduction. This figure shows the accuracy of XGBoost with a feature subset that starting from 60 features (30 features for COIL-20 or MICE), selected by our method and reducing the redundancy according to the correlation between the surrogate representations.