Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Poisson Hyperplane Processes with Rectified Linear Units

Shufei Ge, Shijia Wang, Lloyd Elliott

TL;DR

It is shown that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network and that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions.

Abstract

Neural networks have shown state-of-the-art performances in various classification and regression tasks. Rectified linear units (ReLU) are often used as activation functions for the hidden layers in a neural network model. In this article, we establish the connection between the Poisson hyperplane processes (PHP) and two-layer ReLU neural networks. We show that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network. In addition, we show that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions. Finally, we propose an annealed sequential Monte Carlo algorithm for Bayesian inference. Our numerical experiments demonstrate that our proposed method outperforms the classic two-layer ReLU neural network. The implementation of our proposed model is available at https://github.com/ShufeiGe/Pois_Relu.git.

Poisson Hyperplane Processes with Rectified Linear Units

TL;DR

It is shown that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network and that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions.

Abstract

Neural networks have shown state-of-the-art performances in various classification and regression tasks. Rectified linear units (ReLU) are often used as activation functions for the hidden layers in a neural network model. In this article, we establish the connection between the Poisson hyperplane processes (PHP) and two-layer ReLU neural networks. We show that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network. In addition, we show that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions. Finally, we propose an annealed sequential Monte Carlo algorithm for Bayesian inference. Our numerical experiments demonstrate that our proposed method outperforms the classic two-layer ReLU neural network. The implementation of our proposed model is available at https://github.com/ShufeiGe/Pois_Relu.git.
Paper Structure (14 sections, 3 theorems, 16 equations, 11 figures, 6 tables, 2 algorithms)

This paper contains 14 sections, 3 theorems, 16 equations, 11 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Basic Notations
  4. Two-layer ReLU Neural Networks
  5. The Poisson Hyperplane Processes
  6. From Poisson hyperplane processes to two-layer ReLU neural networks
  7. Decomposition Propositions
  8. Model and Inference
  9. Model
  10. Inference
  11. Simulation
  12. Application
  13. Discussion
  14. Additional numerical results

Key Result

proposition thmcounterproposition

If $\boldsymbol{P}_{\lambda}(\mathbb{D})$ is a PPP with $s-$finite intensity measure $\lambda$ on $(\mathbb{D},\mathcal{D})$, then it can be decomposed into a finite number, denoted as $K$, of independent PPP with $s-$finite intensity measure $\lambda_i$ on $\mathbb{D}$, such that $\sum_{i=1}^{K}\la

Figures (11)

  • Figure 1: A PHP representation of a two-layer Bayesian ReLU neural network with Gaussian priors on weights.
  • Figure 2: Visualization of $10$ randomly selected simulated datasets in the first simulation study. The horizontal-axis and vertical-axis are values of dots' coordinates. Dot color indicates the value of response $y$, and the black lines represent the corresponding generated lines.
  • Figure 3: RMSE across different methods on train and test data splits generated in the first simulation study. The solid boxes (lines) represent the RMSE on the train datasets and the dotted ones indicate the values with respect to the test datasets. Left) The RMSE of our proposed method with varying $L$ and $R$. Right) Classical machine learning methods and our proposed model ($L=1000,~R=100$). The number of hyperplanes of our proposed methods is fixed to $2$ in these experiments.
  • Figure 4: Left) Mean RMSE across different methods on train and test data splits when $|\boldsymbol{P}|=5,~10$ and $p$ varies in the second simulation study. The solid lines represent the RMSE on the train datasets and the dotted ones indicate the values with respect to the test datasets. Right) Running time (in minutes) of our proposed methods with different number of CPUs when $p=20$, $|\boldsymbol{P}|=5$, $L=1000$, and $R=100$. All experiments were run on Digital Alliance Canada's cedar cluster, with memory size set to 8G for each job.
  • Figure 5: Left) The RMSE of the two decomposition approaches compared to the whole model on train and test data from $100$ random experiments for the third simulation study. Right) Running time ratios for decomposition approaches of every single job compared to the whole model. The baseline is the running time for every single job of the whole model. All experiments were run on Digital Alliance Canada's cedar cluster, with memory size set to 8G for each job.
  • ...and 6 more figures

Theorems & Definitions (6)

  • proposition thmcounterproposition
  • proof
  • proposition thmcounterproposition
  • proof
  • proposition thmcounterproposition
  • proof