Backpropagation Neural Tree

TL;DR

This work introduces Backpropagation Neural Tree (BNeuralT), a stochastic, dendritic-tree–like single-neuron model trained via recursive backpropagation using forward post-order and backward pre-order passes. By constructing ad hoc $m$-ary trees with leaves as inputs and internal nodes applying dendritic nonlinearities, BNeuralT delivers highly sparse yet competitive performance across classification, regression, and pattern recognition tasks, often with far fewer parameters than standard MLPs. Empirical results show RMSprop-enabled BNeuralT achieving average classification accuracy of $89.1\%$ (vs. $86.8\%$ for MLP) and substantially smaller model sizes, while MNIST experiments demonstrate competitive performance relative to tree-based methods on raw pixel inputs. The paper argues that such biologically plausible, parsimonious architectures offer a path toward efficient, interpretable, and scalable learning, with a robust convergence profile across multiple SGD optimizers.

Abstract

We propose a novel algorithm called Backpropagation Neural Tree (BNeuralT), which is a stochastic computational dendritic tree. BNeuralT takes random repeated inputs through its leaves and imposes dendritic nonlinearities through its internal connections like a biological dendritic tree would do. Considering the dendritic-tree like plausible biological properties, BNeuralT is a single neuron neural tree model with its internal sub-trees resembling dendritic nonlinearities. BNeuralT algorithm produces an ad hoc neural tree which is trained using a stochastic gradient descent optimizer like gradient descent (GD), momentum GD, Nesterov accelerated GD, Adagrad, RMSprop, or Adam. BNeuralT training has two phases, each computed in a depth-first search manner: the forward pass computes neural tree's output in a post-order traversal, while the error backpropagation during the backward pass is performed recursively in a pre-order traversal. A BNeuralT model can be considered a minimal subset of a neural network (NN), meaning it is a "thinned" NN whose complexity is lower than an ordinary NN. Our algorithm produces high-performing and parsimonious models balancing the complexity with descriptive ability on a wide variety of machine learning problems: classification, regression, and pattern recognition.

Figures (19)

• Figure 1: Neural Trees: (a) A neural tree example of a three-class classification learning problem. The root node $v^3_0$ takes three immediate children: $v_1, v_3,$ and $v_4$, each respectively designated to a class $c_1, c_2,$ and $c_3$. The internal nodes (shaded in gray) are neural nodes and take an activation function $\varphi(\cdot)$ and leaf nodes are inputs. Each designated output class has its subtree. This tree takes its input from the set {$x_1, x_2, \ldots, x_5$}. The link $w_i^{v_j}$ between nodes are neural weights. (b) A neural tree example for a regression problem has one output node $v_0$.
• Figure 2: Biologically plausible neural computation using dendritic trees. Red circle represents a neuron (soma), black lines are dendrites, and the numbers indicate inputs.
• Figure 3: Forward pass (left) and backward pass (right) computation. The arrows show the direction of computation.
• Figure 4: Left. Backpropagation Neural Tree. Output node $v_k$ yields output $y$ using forward pass upon receiving inputs $x_i$ from leaf nodes. Each node is linked with an edge weight $w_{ij}$. The backward pass propagates error $e = (y - \hat{y})$ back to input nodes to compute weight change $\Delta w$. Right. Backpropagation of error from an output node, $k$; to a hidden node, $j$; to an input node, $i$; and to bias inputs $b_k$ and $b_j$. Dashed lines represent error backpropagation and computation $\delta$ and gradient $\nabla w$ (cf. Algorithm \ref{['algo:gradient_compute']}) to find weight change $\Delta w$ that help stochastic gradient descent (cf. Algorithm \ref{['algo:sgd']}).
• Figure 5: Classification Trees. (a) - (i) show test accuracy and tree size of select high performing trees of datasets. The black node in a tree is its root node, class output nodes are in red, function nodes are in blue, and leaf nodes are in green. The link connecting nodes are neural weights.