Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning using Linear Support Vector Machines

Yichuan Tang

TL;DR

The paper questions the default use of softmax cross-entropy in deep classifiers and proposes a linear L2-SVM top layer that can be trained end-to-end via backpropagation. Across MNIST, CIFAR-10, and a facial expression recognition task, the DLSVM top layer yields consistent accuracy gains, attributed to the margin-based regularization of the SVM loss rather than optimization tweaks alone. The results show notable improvements (e.g., MNIST 0.87% vs 0.99% error; CIFAR-10 11.9% vs 14.0% error) and competitive facial expression recognition scores, suggesting that switching to an SVM top layer is a simple, effective alternative for discriminative deep models. The authors also analyze the regularization vs optimization dynamics, indicating that the margin-based objective plays a key role in the observed gains.

Abstract

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.

Deep Learning using Linear Support Vector Machines

TL;DR

The paper questions the default use of softmax cross-entropy in deep classifiers and proposes a linear L2-SVM top layer that can be trained end-to-end via backpropagation. Across MNIST, CIFAR-10, and a facial expression recognition task, the DLSVM top layer yields consistent accuracy gains, attributed to the margin-based regularization of the SVM loss rather than optimization tweaks alone. The results show notable improvements (e.g., MNIST 0.87% vs 0.99% error; CIFAR-10 11.9% vs 14.0% error) and competitive facial expression recognition scores, suggesting that switching to an SVM top layer is a simple, effective alternative for discriminative deep models. The authors also analyze the regularization vs optimization dynamics, indicating that the margin-based objective plays a key role in the observed gains.

Abstract

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.

Paper Structure

This paper contains 13 sections, 11 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The model
  3. Softmax
  4. Support Vector Machines
  5. Multiclass SVMs
  6. Deep Learning with Support Vector Machines
  7. Experiments
  8. Facial Expression Recognition
  9. Softmax vs. DLSVM
  10. MNIST
  11. CIFAR-10
  12. Regularization or Optimization
  13. Conclusions

Figures (4)

  • Figure 1: Training data. Each column consists of faces of the same expression: starting from the leftmost column: Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral.
  • Figure 2: Cross validation performance of the two models. Result is averaged over 8 folds.
  • Figure 3: Filters from convolutional net with softmax.
  • Figure 4: Filters from convolutional net with L2-SVM.