Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using SVM to Estimate and Predict Binary Choice Models

Yoosoon Chang, Joon Y. Park, Guo Yan

TL;DR

This paper analyzes using support vector machines to estimate binary choice models (BCMs). It proves that, under a linear conditional mean condition on covariates, the SVM slope consistently estimates the BCM slope up to a positive scalar and is asymptotically equivalent to logistic regression in that sense, though SVM is not a QMLE. The intercept can be consistently estimated once a consistent slope is obtained, via a second-step maximum score approach, and class imbalance is shown to threaten SVM consistency unless class weights are used. The results are complemented by finite-sample simulations showing that, while SVM and logistic regression have similar asymptotic properties, their small-sample performance can differ across designs; class-weighted SVM effectively mitigates imbalance effects. Overall, the work clarifies the interpretability of SVM for BCMs and provides practical guidance on when and how to apply it, including the importance of weighting to handle imbalance.

Abstract

The support vector machine (SVM) has an asymptotic behavior that parallels that of the quasi-maximum likelihood estimator (QMLE) for binary outcomes generated by a binary choice model (BCM), although it is not a QMLE. We show that, under the linear conditional mean condition for covariates given the systematic component used in the QMLE slope consistency literature, the slope of the separating hyperplane given by the SVM consistently estimates the BCM slope parameter, as long as the class weight is used as required when binary outcomes are severely imbalanced. The SVM slope estimator is asymptotically equivalent to that of logistic regression in this sense. The finite-sample performance of the two estimators can be quite distinct depending on the distributions of covariates and errors, but neither dominates the other. The intercept parameter of the BCM can be consistently estimated once a consistent estimator of its slope parameter is obtained.

Using SVM to Estimate and Predict Binary Choice Models

TL;DR

This paper analyzes using support vector machines to estimate binary choice models (BCMs). It proves that, under a linear conditional mean condition on covariates, the SVM slope consistently estimates the BCM slope up to a positive scalar and is asymptotically equivalent to logistic regression in that sense, though SVM is not a QMLE. The intercept can be consistently estimated once a consistent slope is obtained, via a second-step maximum score approach, and class imbalance is shown to threaten SVM consistency unless class weights are used. The results are complemented by finite-sample simulations showing that, while SVM and logistic regression have similar asymptotic properties, their small-sample performance can differ across designs; class-weighted SVM effectively mitigates imbalance effects. Overall, the work clarifies the interpretability of SVM for BCMs and provides practical guidance on when and how to apply it, including the importance of weighting to handle imbalance.

Abstract

The support vector machine (SVM) has an asymptotic behavior that parallels that of the quasi-maximum likelihood estimator (QMLE) for binary outcomes generated by a binary choice model (BCM), although it is not a QMLE. We show that, under the linear conditional mean condition for covariates given the systematic component used in the QMLE slope consistency literature, the slope of the separating hyperplane given by the SVM consistently estimates the BCM slope parameter, as long as the class weight is used as required when binary outcomes are severely imbalanced. The SVM slope estimator is asymptotically equivalent to that of logistic regression in this sense. The finite-sample performance of the two estimators can be quite distinct depending on the distributions of covariates and errors, but neither dominates the other. The intercept parameter of the BCM can be consistently estimated once a consistent estimator of its slope parameter is obtained.
Paper Structure (25 sections, 12 theorems, 158 equations, 1 figure, 2 tables)

This paper contains 25 sections, 12 theorems, 158 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Support Vector Machine
  4. Binary Choice Model
  5. Classification of BCM Outcomes by SVM
  6. Asymptotics for SVM Estimator
  7. Asymptotic Framework
  8. Gradient and Hessian for SVM when well-defined
  9. Consistency of SVM Estimator
  10. Estimation of Intercept Given Consistent SVM Slope Estimator
  11. Asymptotic Distributions
  12. SVM and QMLE
  13. Noninterpretability of SVM as QMLE
  14. Differing Effects of Imbalance
  15. Illustration: Imbalance and the SVM Slope Consistency
  16. ...and 10 more sections

Key Result

Lemma 3.1

Let Assumption aspn-regularity hold. Then $\hat{\theta} \to_p \theta_\ast$.

Figures (1)

  • Figure 1: Existence of $c_\ast, r_\ast$ under $V=_d \mathbb N(\mu,1), U=_d \mathbb N(0,1)$

Theorems & Definitions (21)

  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Theorem 3.4
  • Corollary 3.5
  • Proposition 3.6
  • Theorem 3.7
  • Lemma 3.8
  • Lemma 3.9
  • Theorem 3.10
  • ...and 11 more