SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

TL;DR

SmartMixed addresses the inefficiency of uniform activation functions by enabling per-neuron activation learning from a fixed pool using a two-phase training scheme. Phase 1 employs differentiable hard Gumbel–Softmax sampling to let each neuron select an activation, while Phase 2 fixes these choices to form a computationally efficient network that continues training. Evaluated on MNIST across 18 architectures, SmartMixed consistently ranks in the top-3 and reveals clear depth-dependent activation preferences: early layers favor ReLU/LeakyReLU and deeper layers favor ELU/SELU. The approach delivers activation specialization without sacrificing inference efficiency, offering practical insights for activation design and neural architecture optimization.

Abstract

The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a two-phase training strategy that allows networks to learn optimal per-neuron activation functions while preserving computational efficiency at inference. In the first phase, neurons adaptively select from a pool of candidate activation functions (ReLU, Sigmoid, Tanh, Leaky ReLU, ELU, SELU) using a differentiable hard-mixture mechanism. In the second phase, each neuron's activation function is fixed according to the learned selection, resulting in a computationally efficient network that supports continued training with optimized vectorized operations. We evaluate SmartMixed on the MNIST dataset using feedforward neural networks of varying depths. The analysis shows that neurons in different layers exhibit distinct preferences for activation functions, providing insights into the functional diversity within neural architectures.

Figures (6)

• Figure 1: Training and validation loss curves during Phase 1 (selective training).
• Figure 2: Training and validation loss curves during Phase 2 (mixed network training) showing continued training for 350 epochs.
• Figure 3: Final activation function distribution across layers after Phase 1 training. Early layers favor ReLU and Leaky ReLU, while deeper layers increasingly prefer ELU and SELU activation functions.
• Figure 4: Mean Reciprocal Rank analysis across all architectures. Leaky ReLU demonstrates the highest MRR, followed closely by ReLU and SmartMixed. SmartMixed shows consistent performance by ranking in top positions across diverse architectures.
• Figure 5: Ranking distribution showing the frequency of each rank position (1st through 7th) achieved by each activation function across all 18 architectures. SmartMixed demonstrates consistent performance with frequent appearances in top positions.