Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection

Pouya Shaeri, Ariane Middel

TL;DR

MID-L addresses the inefficiency of activating all neurons by introducing a per-input, differentiable Top-$k$ gating that interpolates between a lightweight path and a full-capacity path. It is plug-and-play and architecture-agnostic, enabling dynamic, per-neuron sparsity during inference. Across vision, tabular, and text benchmarks, MID-L achieves substantial reductions in active neurons (ANR) and FLOPs while maintaining or improving accuracy, and SMI analyses confirm the selected neurons are highly informative. The approach demonstrates robustness under data corruption and noisy labels, with favorable latency and memory profiles, offering a practical route to compute-aware, data-efficient neural networks.

Abstract

Modern neural networks often activate all neurons for every input, leading to unnecessary computation and inefficiency. We introduce Matrix-Interpolated Dropout Layer (MID-L), a novel module that dynamically selects and activates only the most informative neurons by interpolating between two transformation paths via a learned, input-dependent gating vector. Unlike conventional dropout or static sparsity methods, MID-L employs a differentiable Top-k masking strategy, enabling per-input adaptive computation while maintaining end-to-end differentiability. MID-L is model-agnostic and integrates seamlessly into existing architectures. Extensive experiments on six benchmarks, including MNIST, CIFAR-10, CIFAR-100, SVHN, UCI Adult, and IMDB, show that MID-L achieves up to average 55\% reduction in active neurons, 1.7$\times$ FLOPs savings, and maintains or exceeds baseline accuracy. We further validate the informativeness and selectivity of the learned neurons via Sliced Mutual Information (SMI) and observe improved robustness under overfitting and noisy data conditions. Additionally, MID-L demonstrates favorable inference latency and memory usage profiles, making it suitable for both research exploration and deployment on compute-constrained systems. These results position MID-L as a general-purpose, plug-and-play dynamic computation layer, bridging the gap between dropout regularization and efficient inference.

MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection

TL;DR

MID-L addresses the inefficiency of activating all neurons by introducing a per-input, differentiable Top- gating that interpolates between a lightweight path and a full-capacity path. It is plug-and-play and architecture-agnostic, enabling dynamic, per-neuron sparsity during inference. Across vision, tabular, and text benchmarks, MID-L achieves substantial reductions in active neurons (ANR) and FLOPs while maintaining or improving accuracy, and SMI analyses confirm the selected neurons are highly informative. The approach demonstrates robustness under data corruption and noisy labels, with favorable latency and memory profiles, offering a practical route to compute-aware, data-efficient neural networks.

Abstract

Modern neural networks often activate all neurons for every input, leading to unnecessary computation and inefficiency. We introduce Matrix-Interpolated Dropout Layer (MID-L), a novel module that dynamically selects and activates only the most informative neurons by interpolating between two transformation paths via a learned, input-dependent gating vector. Unlike conventional dropout or static sparsity methods, MID-L employs a differentiable Top-k masking strategy, enabling per-input adaptive computation while maintaining end-to-end differentiability. MID-L is model-agnostic and integrates seamlessly into existing architectures. Extensive experiments on six benchmarks, including MNIST, CIFAR-10, CIFAR-100, SVHN, UCI Adult, and IMDB, show that MID-L achieves up to average 55\% reduction in active neurons, 1.7 FLOPs savings, and maintains or exceeds baseline accuracy. We further validate the informativeness and selectivity of the learned neurons via Sliced Mutual Information (SMI) and observe improved robustness under overfitting and noisy data conditions. Additionally, MID-L demonstrates favorable inference latency and memory usage profiles, making it suitable for both research exploration and deployment on compute-constrained systems. These results position MID-L as a general-purpose, plug-and-play dynamic computation layer, bridging the gap between dropout regularization and efficient inference.
Paper Structure (34 sections, 15 equations, 4 figures, 12 tables)

This paper contains 34 sections, 15 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Block Architecture
  5. Formulation
  6. Fully Connected Layer
  7. Backward pass.
  8. Fully Connected Layer with Dropout
  9. Backward pass.
  10. MID-L: Matrix-Interpolated Dropout Layer
  11. Forward pass.
  12. Backward pass.
  13. Implementation Notes
  14. Experiments and Results
  15. Datasets
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Overview of our proposed MID-L framework. The workflow includes integration into standard neural networks, dynamically selects informative neurons, and is evaluated across multiple benchmarks.
  • Figure 2: MID-L Architecture
  • Figure 3: Visualization of Sliced Mutual Information (SMI) vs activation frequency for neurons selected by MID-L. On both MNIST and CIFAR-10, MID-L achieves a favorable balance of informativeness and sparsity, with selective activation of the most informative neurons.
  • Figure 4: t-SNE visualization of last layer embeddings from MID-L on different datasets. MID-L shows clear separability on MNIST, with more entangled clusters on CIFAR-10 and SVHN.