Local Control Networks (LCNs): Optimizing Flexibility in Neural Network Data Pattern Capture

TL;DR

The paper tackles the rigidity of fixed activation functions in traditional MLPs by introducing Local Control Networks (LCNs), which assign learnable B-spline activations to each neuron, enabling localized, diverse activation curves. It provides a theoretical analysis of gradient behavior, robustness, and learning efficiency, and demonstrates through experiments that LCNs offer competitive or superior performance to MLPs and KANs across basic ML, vision, and symbolic tasks, with faster convergence and lower computational load. The key contributions are the per-neuron spline activation framework, the demonstration of localized gradient updates as a natural regularizer, and empirical evidence of efficiency and scalability advantages over KANs. Practically, LCNs present a simpler, more scalable alternative to KANs that retains strong expressiveness while improving interpretability and training stability, making them suitable for a wide range of real-world applications where activation diversity matters.

Abstract

The widespread use of Multi-layer perceptrons (MLPs) often relies on a fixed activation function (e.g., ReLU, Sigmoid, Tanh) for all nodes within the hidden layers. While effective in many scenarios, this uniformity may limit the networks ability to capture complex data patterns. We argue that employing the same activation function at every node is suboptimal and propose leveraging different activation functions at each node to increase flexibility and adaptability. To achieve this, we introduce Local Control Networks (LCNs), which leverage B-spline functions to enable distinct activation curves at each node. Our mathematical analysis demonstrates the properties and benefits of LCNs over conventional MLPs. In addition, we demonstrate that more complex architectures, such as Kolmogorov-Arnold Networks (KANs), are unnecessary in certain scenarios, and LCNs can be a more efficient alternative. Empirical experiments on various benchmarks and datasets validate our theoretical findings. In computer vision tasks, LCNs achieve marginal improvements over MLPs and outperform KANs by approximately 5\%, while also being more computationally efficient than KANs. In basic machine learning tasks, LCNs show a 1\% improvement over MLPs and a 0.6\% improvement over KANs. For symbolic formula representation tasks, LCNs perform on par with KANs, with both architectures outperforming MLPs. Our findings suggest that diverse activations at the node level can lead to improved performance and efficiency.

Figures (7)

• Figure 1: Limitations of conventional DNNs and the corresponding improvements from LCNs.
• Figure 2: Local Control Network Architecture with B-spline Activation Functions.
• Figure 3: Comparison of accuracy over the number of parameters for MLP, KAN, and LCN models on four datasets.
• Figure 4: Comparison of accuracy over the number of parameters for MLP, KAN, and LCN models on MNIST and FMNIST datasets.
• Figure 5: Comparison of accuracy over the number of parameters for MLP, KAN, and LCN models on symbolic functions.