Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ray-Tracing for Conditionally Activated Neural Networks

Claudio Gallicchio, Giuseppe Nuti

TL;DR

RayTracing addresses the computational cost of large Mixture of Experts models by proposing a hierarchical MoE architecture where blocks are activated via a thresholded firing rate $r^{(i)}$ and a network-wide threshold $\theta$ that is progressively lowered to unfold the architecture. Each block computes $\mathbf{s}^{(i)} = \text{ReLU}(r^{(i)} - \theta)\; \text{Softmax}(\mathcal{F}^{(i)}(\mathbf{z}^{(i)}))$, and backpropagation proceeds through the activated paths, enabling path-specific training. The approach yields competitive accuracy with substantial parameter reductions (over 50% on average) on datasets including CIFAR-10, without requiring auxiliary load-balancing penalties, and reveals that more difficult inputs activate more network blocks. This work has practical implications for efficient, real-time inference in time-sensitive domains and suggests avenues for hardware-friendly scalable conditional computation. It also opens potential extensions to sequence learning and reservoir/state-space approaches for further efficiency gains.

Abstract

In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.

Ray-Tracing for Conditionally Activated Neural Networks

TL;DR

RayTracing addresses the computational cost of large Mixture of Experts models by proposing a hierarchical MoE architecture where blocks are activated via a thresholded firing rate and a network-wide threshold that is progressively lowered to unfold the architecture. Each block computes , and backpropagation proceeds through the activated paths, enabling path-specific training. The approach yields competitive accuracy with substantial parameter reductions (over 50% on average) on datasets including CIFAR-10, without requiring auxiliary load-balancing penalties, and reveals that more difficult inputs activate more network blocks. This work has practical implications for efficient, real-time inference in time-sensitive domains and suggests avenues for hardware-friendly scalable conditional computation. It also opens potential extensions to sequence learning and reservoir/state-space approaches for further efficiency gains.

Abstract

In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.

Paper Structure

This paper contains 4 sections, 1 equation, 3 figures.

Table of Contents

  1. Introduction & Background
  2. Ray-Tracing Neural Networks
  3. Experiments
  4. Conclusions

Figures (3)

  • Figure 1: (a) Each of the stacked blocks (experts) is activated when at least one of the incoming gates' value is above a threshold, $\theta$ (set at network level). (b) To sequentially sample approximations to inference problem, we start with a large value of $\theta$, progressively decreasing it (thus activating an increasing number of blocks) until a stopping condition has been met.
  • Figure 2: Scatter plot illustrating the trade-off between accuracy (evaluated on the test set) and the number of effectively used parameters (evaluated at inference time on the test set). Different colors correspond to different datasets. Marker $\medblackcircle$ corresponds to Raytracing, $\medblacktriangleup$ corresponds to the baseline MLP. Points higher up in the plot correspond to better accuracy, points further to the left indicate greater computational efficiency. For RayTracing, we report the average number of used parameters across the entire test set.
  • Figure 3: Input samples activating different proportions of the RayTracing neural network (indicated as a percentage value on top of each image). For each dataset, we present the 12 images that activated the least number of blocks (respective top rows) and the most number of blocks (bottom rows).