Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Sigmoid-MSE Great Again: Output Reset Challenges Softmax Cross-Entropy in Neural Network Classification

Kanishka Tyagi, Chinmay Rane, Ketaki Vaidya, Jeshwanth Challgundla, Soumitro Swapan Auddy, Michael Manry

TL;DR

The findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.

Abstract

This study presents a comparative analysis of two objective functions, Mean Squared Error (MSE) and Softmax Cross-Entropy (SCE) for neural network classification tasks. While SCE combined with softmax activation is the conventional choice for transforming network outputs into class probabilities, we explore an alternative approach using MSE with sigmoid activation. We introduce the Output Reset algorithm, which reduces inconsistent errors and enhances classifier robustness. Through extensive experiments on benchmark datasets (MNIST, CIFAR-10, and Fashion-MNIST), we demonstrate that MSE with sigmoid activation achieves comparable accuracy and convergence rates to SCE, while exhibiting superior performance in scenarios with noisy data. Our findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.

Making Sigmoid-MSE Great Again: Output Reset Challenges Softmax Cross-Entropy in Neural Network Classification

TL;DR

The findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.

Abstract

This study presents a comparative analysis of two objective functions, Mean Squared Error (MSE) and Softmax Cross-Entropy (SCE) for neural network classification tasks. While SCE combined with softmax activation is the conventional choice for transforming network outputs into class probabilities, we explore an alternative approach using MSE with sigmoid activation. We introduce the Output Reset algorithm, which reduces inconsistent errors and enhances classifier robustness. Through extensive experiments on benchmark datasets (MNIST, CIFAR-10, and Fashion-MNIST), we demonstrate that MSE with sigmoid activation achieves comparable accuracy and convergence rates to SCE, while exhibiting superior performance in scenarios with noisy data. Our findings indicate that MSE, despite its traditional association with regression tasks, serves as a viable alternative for classification problems, challenging conventional wisdom about neural network training strategies.

Paper Structure

This paper contains 16 sections, 1 theorem, 31 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Structure and Notation
  4. Regression-Based Classifier
  5. Mathematical Background
  6. Problems With MSE Type Training of Neural Networks
  7. The Potential for MSE-Based Optimal Classifiers
  8. Solution for Problem P1 and $N_c = 2$
  9. Solution for Problem P3
  10. Solution for P1, P2 and P4
  11. Enhancements in Linear Classifier Design
  12. Target Adjustment using Output Reset
  13. Experimental Results
  14. Discussion
  15. Conclusion
  16. ...and 1 more sections

Key Result

Theorem 1

$\mathbf{W}_{\text{opt}}$ is the solution to a least squares problem.

Figures (6)

  • Figure 1: Linear classifier
  • Figure 2: MSE training example with $N = M = N_c = 2$, showing class boundary, optimal hyperplane, and support vectors. Squares and circles represent class patterns, with errors indicated by lines perpendicular to the boundary.
  • Figure 3: Average Testing Predictive Error for SCE, MSE-OR, and SMSE-OR Algorithms Across Datasets
  • Figure 4: Best Average Validation Iteration for SCE, MSE-OR, and SMSE-OR Algorithms Across Datasets
  • Figure 5: Average Testing Predictive Error Across Algorithms and Datasets
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 1