Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature Selection and Regularization in Multi-Class Classification: An Empirical Study of One-vs-Rest Logistic Regression with Gradient Descent Optimization and L1 Sparsity Constraints

Jahidul Arafat, Fariha Tasmin, Sanjaya Poudel

TL;DR

This study evaluates One-vs-Rest logistic regression on the UCI Wine dataset to balance accuracy, feature sparsity, and deployment practicality. By comparing from-scratch gradient descent with scikit-learn solvers and examining L1 regularization, the work reveals strong class-specific chemical signatures and actionable feature subsets. The key findings show that sparse models retain substantial accuracy (54–69% feature reduction with around 4.6% average accuracy loss) and that a 5-feature subset can achieve near-desired performance with significant cost and time savings, enabling real-time production decisions. The results inform deployment decisions in analytical chemistry, highlighting the value of interpretable models, robust feature selection, and cost-aware trade-offs for varietal authentication across resource-constrained lab environments.

Abstract

Multi-class wine classification presents fundamental trade-offs between model accuracy, feature dimensionality, and interpretability - critical factors for production deployment in analytical chemistry. This paper presents a comprehensive empirical study of One-vs-Rest logistic regression on the UCI Wine dataset (178 samples, 3 cultivars, 13 chemical features), comparing from-scratch gradient descent implementation against scikit-learn's optimized solvers and quantifying L1 regularization effects on feature sparsity. Manual gradient descent achieves 92.59 percent mean test accuracy with smooth convergence, validating theoretical foundations, though scikit-learn provides 24x training speedup and 98.15 percent accuracy. Class-specific analysis reveals distinct chemical signatures with heterogeneous patterns where color intensity varies dramatically (0.31 to 16.50) across cultivars. L1 regularization produces 54-69 percent feature reduction with only 4.63 percent accuracy decrease, demonstrating favorable interpretability-performance trade-offs. We propose an optimal 5-feature subset achieving 62 percent complexity reduction with estimated 92-94 percent accuracy, enabling cost-effective deployment with 80 dollars savings per sample and 56 percent time reduction. Statistical validation confirms robust generalization with sub-2ms prediction latency suitable for real-time quality control. Our findings provide actionable guidelines for practitioners balancing comprehensive chemical analysis against targeted feature measurement in resource-constrained environments.

Feature Selection and Regularization in Multi-Class Classification: An Empirical Study of One-vs-Rest Logistic Regression with Gradient Descent Optimization and L1 Sparsity Constraints

TL;DR

This study evaluates One-vs-Rest logistic regression on the UCI Wine dataset to balance accuracy, feature sparsity, and deployment practicality. By comparing from-scratch gradient descent with scikit-learn solvers and examining L1 regularization, the work reveals strong class-specific chemical signatures and actionable feature subsets. The key findings show that sparse models retain substantial accuracy (54–69% feature reduction with around 4.6% average accuracy loss) and that a 5-feature subset can achieve near-desired performance with significant cost and time savings, enabling real-time production decisions. The results inform deployment decisions in analytical chemistry, highlighting the value of interpretable models, robust feature selection, and cost-aware trade-offs for varietal authentication across resource-constrained lab environments.

Abstract

Multi-class wine classification presents fundamental trade-offs between model accuracy, feature dimensionality, and interpretability - critical factors for production deployment in analytical chemistry. This paper presents a comprehensive empirical study of One-vs-Rest logistic regression on the UCI Wine dataset (178 samples, 3 cultivars, 13 chemical features), comparing from-scratch gradient descent implementation against scikit-learn's optimized solvers and quantifying L1 regularization effects on feature sparsity. Manual gradient descent achieves 92.59 percent mean test accuracy with smooth convergence, validating theoretical foundations, though scikit-learn provides 24x training speedup and 98.15 percent accuracy. Class-specific analysis reveals distinct chemical signatures with heterogeneous patterns where color intensity varies dramatically (0.31 to 16.50) across cultivars. L1 regularization produces 54-69 percent feature reduction with only 4.63 percent accuracy decrease, demonstrating favorable interpretability-performance trade-offs. We propose an optimal 5-feature subset achieving 62 percent complexity reduction with estimated 92-94 percent accuracy, enabling cost-effective deployment with 80 dollars savings per sample and 56 percent time reduction. Statistical validation confirms robust generalization with sub-2ms prediction latency suitable for real-time quality control. Our findings provide actionable guidelines for practitioners balancing comprehensive chemical analysis against targeted feature measurement in resource-constrained environments.
Paper Structure (29 sections, 12 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 29 sections, 12 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Problem Formalization
  4. Methodology
  5. Experimental Design
  6. Experimental Results
  7. RQ1: Gradient Descent vs Scikit-learn Implementation Comparison
  8. RQ2: Class-Specific Feature Importance Patterns
  9. RQ3: L1 Regularization Effects on Sparsity and Performance
  10. RQ4: Optimal Feature Selection for Production Deployment
  11. Model Consistency and Stability Analysis
  12. Comprehensive Performance Summary
  13. Discussion
  14. Model Selection for Production Wine Authentication
  15. Class-Specific Feature Patterns and Chemical Interpretability
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Gradient descent convergence comprehensive analysis. Panel (a) shows training loss trajectories across 10,000 iterations for all three binary classifiers (Class 0 vs Rest in red, Class 1 vs Rest in blue, Class 2 vs Rest in green). Smooth exponential decay without oscillations validates learning rate selection ($\eta=0.0001$) and confirms objective convexity. Final loss values annotated (Class 0: 0.3664, Class 1: 0.4129, Class 2: 0.3498) indicate successful optimization. Panel (b) displays loss decrease rate per iteration on logarithmic scale, revealing consistent convergence dynamics with steepest descent in initial 2,000 iterations followed by asymptotic stabilization. Panel (c) presents convergence statistics table summarizing initial loss, final loss, reduction percentage, and iteration counts, demonstrating 64.5-68.2% loss reduction across all binary problems.
  • Figure 2: Feature importance comprehensive analysis across models and classes. Panel (a) shows absolute weight magnitudes without regularization via grouped bar chart: Class 0 (red bars) distinguished by alcalinity_of_ash (6.71), proline (6.55), flavanoids (5.66); Class 1 (blue bars) by color_intensity (16.50), proline (15.49), ash (8.20); Class 2 (green bars) by color_intensity (7.02), flavanoids (5.22), od280/od315 (3.50). Heterogeneous patterns evident: color_intensity dominant for Classes 1-2 but negligible for Class 0 (0.31); alcalinity_of_ash critical for Class 0 but moderate for others. Panel (b) presents importance heatmap with annotated values revealing magnitude disparities: Class 1 coefficients reach 16.50 (color_intensity) while Class 2 maximum only 7.02, indicating varying feature separability across binary problems. Panel (c) displays L1 sparsity pattern (green=retained, red=eliminated): Class 0 retains 4/13 features (30.8%), Class 1 retains 6/13 (46.2%), Class 2 retains 5/13 (38.5%), with retention counts annotated. Panel (d) shows aggregate importance ranking identifying universal discriminators: color_intensity (23.83), proline (22.16), alcohol (13.82) highlighted in red as optimal 5-feature subset for production deployment achieving 62% complexity reduction.
  • Figure 3: Class-specific feature patterns via radar plot analysis. Panels (a-c) show individual radar plots for each class with normalized feature importance (0-1 scale) on 13 radial axes. Class 0 (panel a, red) exhibits pronounced vertices toward alcalinity (0.50), proline (0.49), flavanoids (0.42) with minimal color intensity (0.02). Class 1 (panel b, blue) displays extreme extension toward color intensity (1.00) and proline (0.91), creating distinctive elongated polygon shape. Class 2 (panel c, green) shows moderate balanced pattern with color intensity (0.52), flavanoids (0.39), and several secondary features. Panel (d) overlays all three classes revealing non-overlapping polygons confirming heterogeneous signatures. Panel (e) presents importance heatmap with warmer colors indicating higher values, visually emphasizing Class 1's extreme coefficients. Panel (f) shows feature discrimination power via variance across classes: color intensity (variance=0.242) and proline (variance=0.180) achieve highest discrimination, while magnesium (0.003) and total phenols (0.002) provide minimal class distinction.
  • Figure 4: Model performance comprehensive comparison across algorithms and regularization settings. Panel (a) shows test accuracy heatmap: gradient descent achieves 86.11-97.22% (mean 92.59%), unregularized sklearn achieves 97.22-100% (mean 98.15%), L1-regularized sklearn achieves 88.89-97.22% (mean 93.52%). Color intensity from red (lower) to green (higher) enables immediate visual comparison. Annotated percentages facilitate quantitative assessment. Panel (b) displays training accuracy heatmap revealing overfitting patterns: gradient descent shows 92.96-96.48% indicating underfitting, sklearn achieves perfect 100% indicating excellent capacity, L1 shows 95.77-98.59% reflecting regularization's controlled capacity reduction. Panel (c) presents test accuracy bar chart comparison enabling direct visual magnitude comparison across models and classes. Panel (d) plots accuracy-sparsity trade-off in 2D space with point size proportional to model type: rightmost points (100% retention) cluster at high accuracy, leftmost points (30.8-46.2% retention) achieve slightly lower accuracy. L1 configurations occupy favorable region achieving 54-69% feature reduction with only 4.63% mean accuracy decrease (98.15% to 93.52%), demonstrating excellent interpretability-performance balance. Green shaded region indicates high-quality zone (accuracy$>$0.90, retention$>$30%). Annotations show specific retention percentages and accuracy values for all configurations.
  • Figure 5: Comprehensive confusion matrix analysis across all models and classes. Nine panels arranged in 3$\times$3 grid showing gradient descent (rows 1), sklearn no regularization (row 2), and sklearn L1 regularization (row 3) for Classes 0, 1, and 2 (columns). Each matrix displays 2$\times$2 layout with true negatives (top-left), false positives (top-right), false negatives (bottom-left), and true positives (bottom-right). Color intensity indicates count magnitude with annotated values. Titles show accuracy, precision, recall, and F1-score for each binary classifier. Panel letters (a-i) enable cross-referencing. Gradient descent achieves 86.11-97.22% accuracy with 0-4 errors; sklearn no-reg achieves 97.22-100% with 0-1 errors; sklearn L1 achieves 88.89-97.22% with 1-4 errors. Class 2 demonstrates strongest performance across all models reflecting superior feature separability. Error patterns reveal false negatives dominate for Class 2 gradient descent (4 missed), while false positives and negatives balance for Class 1 L1 regularization.
  • ...and 2 more figures