Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Experiment on Feature Selection using Logistic Regression

Raisa Islam, Subhasish Mazumdar, Rakibul Islam

TL;DR

The paper addresses feature selection for classification by synthesizing $L1$ and $L2$ regularization in logistic regression to rank features and then using the intersection of these rankings as a compact feature set. On the CIC-IDS2018 dataset, a 22-feature common subset achieves near-baseline accuracy for LR models while dramatically reducing dimensionality (72% reduction) and also preserves high accuracy for Random Forest and Decision Tree classifiers. Including a problematic class degrades linear models more than ensemble methods, and detailed confusion-matrix analyses reveal tradeoffs in identifying the problematic versus confounding classes. The approach offers a practical, regularization-driven path to compact feature sets with broad applicability, though validation on additional datasets is warranted and a heuristic for feature-count selection is proposed for future work.

Abstract

In supervised machine learning, feature selection plays a very important role by potentially enhancing explainability and performance as measured by computing time and accuracy-related metrics. In this paper, we investigate a method for feature selection based on the well-known L1 and L2 regularization strategies associated with logistic regression (LR). It is well known that the learned coefficients, which serve as weights, can be used to rank the features. Our approach is to synthesize the findings of L1 and L2 regularization. For our experiment, we chose the CIC-IDS2018 dataset owing partly to its size and also to the existence of two problematic classes that are hard to separate. We report first with the exclusion of one of them and then with its inclusion. We ranked features first with L1 and then with L2, and then compared logistic regression with L1 (LR+L1) against that with L2 (LR+L2) by varying the sizes of the feature sets for each of the two rankings. We found no significant difference in accuracy between the two methods once the feature set is selected. We chose a synthesis, i.e., only those features that were present in both the sets obtained from L1 and that from L2, and experimented with it on more complex models like Decision Tree and Random Forest and observed that the accuracy was very close in spite of the small size of the feature set. Additionally, we also report on the standard metrics: accuracy, precision, recall, and f1-score.

An Experiment on Feature Selection using Logistic Regression

TL;DR

The paper addresses feature selection for classification by synthesizing and regularization in logistic regression to rank features and then using the intersection of these rankings as a compact feature set. On the CIC-IDS2018 dataset, a 22-feature common subset achieves near-baseline accuracy for LR models while dramatically reducing dimensionality (72% reduction) and also preserves high accuracy for Random Forest and Decision Tree classifiers. Including a problematic class degrades linear models more than ensemble methods, and detailed confusion-matrix analyses reveal tradeoffs in identifying the problematic versus confounding classes. The approach offers a practical, regularization-driven path to compact feature sets with broad applicability, though validation on additional datasets is warranted and a heuristic for feature-count selection is proposed for future work.

Abstract

In supervised machine learning, feature selection plays a very important role by potentially enhancing explainability and performance as measured by computing time and accuracy-related metrics. In this paper, we investigate a method for feature selection based on the well-known L1 and L2 regularization strategies associated with logistic regression (LR). It is well known that the learned coefficients, which serve as weights, can be used to rank the features. Our approach is to synthesize the findings of L1 and L2 regularization. For our experiment, we chose the CIC-IDS2018 dataset owing partly to its size and also to the existence of two problematic classes that are hard to separate. We report first with the exclusion of one of them and then with its inclusion. We ranked features first with L1 and then with L2, and then compared logistic regression with L1 (LR+L1) against that with L2 (LR+L2) by varying the sizes of the feature sets for each of the two rankings. We found no significant difference in accuracy between the two methods once the feature set is selected. We chose a synthesis, i.e., only those features that were present in both the sets obtained from L1 and that from L2, and experimented with it on more complex models like Decision Tree and Random Forest and observed that the accuracy was very close in spite of the small size of the feature set. Additionally, we also report on the standard metrics: accuracy, precision, recall, and f1-score.
Paper Structure (23 sections, 6 figures, 10 tables)

This paper contains 23 sections, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Logistic regression and Regularization
  4. Random Forest
  5. Decision Tree
  6. Performance evaluation parameters
  7. Accuracy
  8. Precision
  9. Recall
  10. F1-score
  11. Experiment
  12. Dataset and pre-processing
  13. Dataset
  14. pre-processing
  15. Experimental setup
  16. ...and 8 more sections

Figures (6)

  • Figure 1: Accuracy vs number of top features from L1 and L2 orderings.
  • Figure 2: Accuracy vs number of top common features ordered as per Table \ref{['tab:common_features']}.
  • Figure 3: Confusion matrix of LR+L1 model.
  • Figure 4: Confusion matrix of LR+L2 model.
  • Figure 5: Confusion matrix of RF model.
  • ...and 1 more figures