Enhancing Diversity in Multi-objective Feature Selection

TL;DR

The paper tackles diversity loss in binary multi-objective feature selection with NSGA-II by introducing Binary Diverse NSGA-II, which combines genuine initialization (Uniform Covering) and last-front reinitialization to explore unseen regions of the search space. Using two objectives—$f_1$ as $Classification\ Error$ and $f_2$ as the $Ratio\ of\ Selected\ Features$—the method demonstrates substantial improvements in hypervolume (HV) on twelve high-dimensional datasets, achieving train HV around $0.97$ and test HV above $0.70$, while producing solutions with far fewer features. The genuine initialization contributes the most to performance gains, with the substitution step providing an additional but smaller boost, and replacements occur on about 11% of the population per generation. The results indicate that targeted diversification can overcome convergence to local optima without increasing computational complexity relative to NSGA-II, with potential applicability to broader binary MO optimization problems beyond feature selection.

Abstract

Feature selection plays a pivotal role in the data preprocessing and model-building pipeline, significantly enhancing model performance, interpretability, and resource efficiency across diverse domains. In population-based optimization methods, the generation of diverse individuals holds utmost importance for adequately exploring the problem landscape, particularly in highly multi-modal multi-objective optimization problems. Our study reveals that, in line with findings from several prior research papers, commonly employed crossover and mutation operations lack the capability to generate high-quality diverse individuals and tend to become confined to limited areas around various local optima. This paper introduces an augmentation to the diversity of the population in the well-established multi-objective scheme of the genetic algorithm, NSGA-II. This enhancement is achieved through two key components: the genuine initialization method and the substitution of the worst individuals with new randomly generated individuals as a re-initialization approach in each generation. The proposed multi-objective feature selection method undergoes testing on twelve real-world classification problems, with the number of features ranging from 2,400 to nearly 50,000. The results demonstrate that replacing the last front of the population with an equivalent number of new random individuals generated using the genuine initialization method and featuring a limited number of features substantially improves the population's quality and, consequently, enhances the performance of the multi-objective algorithm.

Figures (3)

• Figure 1: Comparing HV plots during optimization for the traditional and proposed diverse NSGA-II on some datasets
• Figure 2: Average pairwise Hamming distance plots during optimization for NSGA-II with and without the substitution component
• Figure 3: Sample plots of the ratio of the replaced individuals in each generation of diverse NSGA-II over generations.