Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two Ridge Solutions for the Incremental Broad Learning System on Added Nodes

Hufei Zhu

TL;DR

This work addresses incremental Broad Learning System (BLS) on added nodes by removing the prior restriction that the ridge parameter must tend to zero. It introduces two ridge-based solutions: Chol-1/Chol-2, which compute output weights from the ridge inverse using an extended inverse Cholesky update for Hermitian matrices partitioned into 2x2 blocks, and R-Inv, which updates the ridge inverse directly (extending Greville’s method to finite ridge). The paper provides detailed construction models, learning procedures, complexity analyses, and extensive MNIST/NORB experiments demonstrating improved testing accuracies and substantial speedups over previous methods. The proposed approaches preserve the original BLS structure while enabling robust, scalable incremental updates with better numerical stability, including division- and square-root-free variants for distributed implementations.

Abstract

The original Broad Learning System (BLS) on new added nodes and its existing efficient implementation both assume the ridge parameter lambda -> 0 in the ridge inverse to approximate the generalized inverse, and compute the generalized inverse solution for the output weights. In this paper, we propose two ridge solutions for the output weights in the BLS on added nodes, where lambda -> 0 is no longer assumed, and lambda can be any positive real number. One of the proposed ridge solutions computes the output weights from the inverse Cholesky factor, which is updated efficiently by extending the existing inverse Cholesky factorization. The other proposed ridge solution computes the output weights from the ridge inverse, and updates the ridge inverse by extending the Greville's method that is a classical tool to compute the generalized inverse of partitioned matrices. For the proposed efficient ridge solution based on the inverse Cholesky factor, we also develop another implementation that is numerically more stable when the ridge parameter lambda is very small. The proposed ridge solution based on the ridge inverse and the numerically more stable implementation of the proposed efficient ridge solution require the same complexity as the original BLS and the existing efficient BLS, respectively. Moreover, the speedups of the proposed efficient ridge solution to the original BLS and the existing efficient BLS are 3 and more than 1.67 respectively, when the computational complexities for each update are compared, and the speedups are 1.99 - 2.52 and 1.31 - 1.58, respectively, when the total training time is compared by numerical experiments. On the other hand, our numerical experiments show that both the proposed ridge solutions for BLS achieve better testing accuracies than the original BLS and the existing efficient BLS.

Two Ridge Solutions for the Incremental Broad Learning System on Added Nodes

TL;DR

This work addresses incremental Broad Learning System (BLS) on added nodes by removing the prior restriction that the ridge parameter must tend to zero. It introduces two ridge-based solutions: Chol-1/Chol-2, which compute output weights from the ridge inverse using an extended inverse Cholesky update for Hermitian matrices partitioned into 2x2 blocks, and R-Inv, which updates the ridge inverse directly (extending Greville’s method to finite ridge). The paper provides detailed construction models, learning procedures, complexity analyses, and extensive MNIST/NORB experiments demonstrating improved testing accuracies and substantial speedups over previous methods. The proposed approaches preserve the original BLS structure while enabling robust, scalable incremental updates with better numerical stability, including division- and square-root-free variants for distributed implementations.

Abstract

The original Broad Learning System (BLS) on new added nodes and its existing efficient implementation both assume the ridge parameter lambda -> 0 in the ridge inverse to approximate the generalized inverse, and compute the generalized inverse solution for the output weights. In this paper, we propose two ridge solutions for the output weights in the BLS on added nodes, where lambda -> 0 is no longer assumed, and lambda can be any positive real number. One of the proposed ridge solutions computes the output weights from the inverse Cholesky factor, which is updated efficiently by extending the existing inverse Cholesky factorization. The other proposed ridge solution computes the output weights from the ridge inverse, and updates the ridge inverse by extending the Greville's method that is a classical tool to compute the generalized inverse of partitioned matrices. For the proposed efficient ridge solution based on the inverse Cholesky factor, we also develop another implementation that is numerically more stable when the ridge parameter lambda is very small. The proposed ridge solution based on the ridge inverse and the numerically more stable implementation of the proposed efficient ridge solution require the same complexity as the original BLS and the existing efficient BLS, respectively. Moreover, the speedups of the proposed efficient ridge solution to the original BLS and the existing efficient BLS are 3 and more than 1.67 respectively, when the computational complexities for each update are compared, and the speedups are 1.99 - 2.52 and 1.31 - 1.58, respectively, when the total training time is compared by numerical experiments. On the other hand, our numerical experiments show that both the proposed ridge solutions for BLS achieve better testing accuracies than the original BLS and the existing efficient BLS.

Paper Structure

This paper contains 38 sections, 158 equations, 1 figure, 6 tables, 14 algorithms.

Table of Contents

  1. Introduction
  2. Existing Generalized Inverse Solutions for the BLS with Added Nodes
  3. Ridge Regression Approximation of the Generalized Inverse
  4. Broad Learning Model on Added Nodes
  5. Incremental Learning for the Original BLS (i.e., Orig.)
  6. The Existing Efficient Generalized Inverse Solution for BLS by Inverse Cholesky Factorization (i.e., Exg-E)
  7. Construction Model and Learning Procedure of BLS
  8. Proposed Ridge Solution for BLS by Inverse Cholesky Factorization of a Partitioned Matrix
  9. Efficient Inverse Cholesky Factorization of a Hermitian Matrix Partitioned into $2 \times 2$ Blocks for Proposed Chol-1
  10. Utilizing the Inverse Cholesky Factor to Compute the Ridge Inverse and the Ridge Solution for Proposed Chol-1
  11. A Numerically More Stable Implementation to Update the Inverse Cholesky Factor and the Ridge Solution (i.e., Chol-2)
  12. Construction Model and Learning Procedure of Proposed Chol-1 and Chol-2
  13. Comparison of Existing and Proposed Efficient BLS Algorithms by Inverse Cholesky Factorization
  14. Proposed Ridge Solution for BLS by Updating the Ridge Inverse
  15. The Algorithm to Update the Ridge Inverse and Solution without Using the Inverse Cholesky Factor (i.e., R-Inv)
  16. ...and 23 more sections

Figures (1)

  • Figure 1: The maximum testing accuracies achieved under different ridge parameters on MNIST and NORB datasets.