Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Kernel Logistic Regression with Nyström Approximation: Theoretical Analysis and Application to Discrete Choice Modelling

José Ángel Martín-Baos, Ricardo García-Ródenas, Luis Rodriguez-Benitez, Michel Bierlaire

TL;DR

The paper tackles the scalability of kernel-based discrete choice models by applying Nyström low-rank approximation to Kernel Logistic Regression, enabling training on very large transport datasets. It provides a theoretical characterization of the KLR solution, introduces a Nyström-based training framework with error bounds, and evaluates four landmark-selection strategies (uniform, k-means, and two leverage-score methods) across large-scale datasets. Empirical results show that a k-means Nyström KLR with L-BFGS-B or Adam optimization achieves robust performance on datasets exceeding 200,000 observations while drastically reducing kernel memory from $O(N^2)$ to $O(NC)$. The work demonstrates Nyström-based KLR as a viable, scalable alternative to traditional discrete choice methods, with actionable guidance on landmark selection and optimization in real-world large-scale applications.

Abstract

The application of kernel-based Machine Learning (ML) techniques to discrete choice modelling using large datasets often faces challenges due to memory requirements and the considerable number of parameters involved in these models. This complexity hampers the efficient training of large-scale models. This paper addresses these problems of scalability by introducing the Nyström approximation for Kernel Logistic Regression (KLR) on large datasets. The study begins by presenting a theoretical analysis in which: i) the set of KLR solutions is characterised, ii) an upper bound to the solution of KLR with Nyström approximation is provided, and finally iii) a specialisation of the optimisation algorithms to Nyström KLR is described. After this, the Nyström KLR is computationally validated. Four landmark selection methods are tested, including basic uniform sampling, a k-means sampling strategy, and two non-uniform methods grounded in leverage scores. The performance of these strategies is evaluated using large-scale transport mode choice datasets and is compared with traditional methods such as Multinomial Logit (MNL) and contemporary ML techniques. The study also assesses the efficiency of various optimisation techniques for the proposed Nyström KLR model. The performance of gradient descent, Momentum, Adam, and L-BFGS-B optimisation methods is examined on these datasets. Among these strategies, the k-means Nyström KLR approach emerges as a successful solution for applying KLR to large datasets, particularly when combined with the L-BFGS-B and Adam optimisation methods. The results highlight the ability of this strategy to handle datasets exceeding 200,000 observations while maintaining robust performance.

Scalable Kernel Logistic Regression with Nyström Approximation: Theoretical Analysis and Application to Discrete Choice Modelling

TL;DR

The paper tackles the scalability of kernel-based discrete choice models by applying Nyström low-rank approximation to Kernel Logistic Regression, enabling training on very large transport datasets. It provides a theoretical characterization of the KLR solution, introduces a Nyström-based training framework with error bounds, and evaluates four landmark-selection strategies (uniform, k-means, and two leverage-score methods) across large-scale datasets. Empirical results show that a k-means Nyström KLR with L-BFGS-B or Adam optimization achieves robust performance on datasets exceeding 200,000 observations while drastically reducing kernel memory from to . The work demonstrates Nyström-based KLR as a viable, scalable alternative to traditional discrete choice methods, with actionable guidance on landmark selection and optimization in real-world large-scale applications.

Abstract

The application of kernel-based Machine Learning (ML) techniques to discrete choice modelling using large datasets often faces challenges due to memory requirements and the considerable number of parameters involved in these models. This complexity hampers the efficient training of large-scale models. This paper addresses these problems of scalability by introducing the Nyström approximation for Kernel Logistic Regression (KLR) on large datasets. The study begins by presenting a theoretical analysis in which: i) the set of KLR solutions is characterised, ii) an upper bound to the solution of KLR with Nyström approximation is provided, and finally iii) a specialisation of the optimisation algorithms to Nyström KLR is described. After this, the Nyström KLR is computationally validated. Four landmark selection methods are tested, including basic uniform sampling, a k-means sampling strategy, and two non-uniform methods grounded in leverage scores. The performance of these strategies is evaluated using large-scale transport mode choice datasets and is compared with traditional methods such as Multinomial Logit (MNL) and contemporary ML techniques. The study also assesses the efficiency of various optimisation techniques for the proposed Nyström KLR model. The performance of gradient descent, Momentum, Adam, and L-BFGS-B optimisation methods is examined on these datasets. Among these strategies, the k-means Nyström KLR approach emerges as a successful solution for applying KLR to large datasets, particularly when combined with the L-BFGS-B and Adam optimisation methods. The results highlight the ability of this strategy to handle datasets exceeding 200,000 observations while maintaining robust performance.
Paper Structure (26 sections, 4 theorems, 78 equations, 5 figures, 5 tables)

This paper contains 26 sections, 4 theorems, 78 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions and scope
  3. Structure
  4. Related work
  5. Multinomial Kernel Logistic Regression
  6. Preliminaries
  7. Analysis of the solution set of the KLR
  8. Nyström-based Multinomial Kernel Logistic Regression
  9. Matrix sketching and the Nyström Method
  10. Column selection methods
  11. Nyström KLR error bounds
  12. Training Algorithms for the Nyström-based Kernel Logistic Regression
  13. Calculating the gradient and loss function in Nyström-based KLR
  14. Line search methods
  15. Experimental results
  16. ...and 11 more sections

Key Result

Theorem 3.1

The general solution is expressed as: where $\boldsymbol{\eta}_{\cdot \cdot} \in L(\mathbf{K})^\perp$, $\boldsymbol{\aleph}^*_{\cdot \cdot}\in L(\mathbf{K})$, and $\boldsymbol{\aleph}^*_{\cdot \cdot}$ is the unique solution to the problem (eq:04_estimation_KLR_RNLL) restricted to the subspace $L(\mathbf{K})$: The problem represented by (eq:04_estimation_KLR_RNLL_restricted) will be known as the

Figures (5)

  • Figure 1: Singular values of the kernel matrix $\mathbf{K}$ for the LPMC and NTS datasets
  • Figure 2: Comparison of Nyström sampling techniques on the LPMC dataset
  • Figure 3: Comparison of Nyström sampling techniques on the NTS dataset
  • Figure 4: Comparison of optimisation algorithms for KLR on the LPMC dataset
  • Figure 5: Comparison of optimisation algorithms for KLR on the NTS dataset

Theorems & Definitions (10)

  • Theorem 3.1: Characterisation of the solution set of the training problem
  • Remark 3.1
  • Theorem 4.1: Error bounds in Nyström
  • Lemma Appendix A.1
  • proof
  • Remark Appendix A.1
  • proof
  • Lemma Appendix B.1
  • proof
  • proof