NeuronSeek: On Stability and Expressivity of Task-driven Neurons

TL;DR

This work introduces NS-TD, a differentiable tensor-decomposition framework for discovering task-driven neurons that tailor aggregation functions to specific tasks. It provides a formal super-super-expressiveness guarantee using standard activations, and demonstrates superior stability and competitive performance across synthetic, tabular, and image benchmarks compared with symbolic-regression and neural baselines. Key contributions include a two-stream polynomial-plus-interaction structure, CP-based efficient interactions, a differentiable L0-regularized search with warm-up, and a rigorous theoretical construction linking task-driven aggregation to dense trajectories. Empirical results show NS-TD achieves robust convergence and state-of-the-art or competitive results with lower computational cost, underscoring its potential as a practical component in diverse neural architectures. Code is available at https://github.com/HanyuPei22/NeuronSeek.

Abstract

Drawing inspiration from our human brain that designs different neurons for different tasks, recent advances in deep learning have explored modifying a network's neurons to develop so-called task-driven neurons. Prototyping task-driven neurons (referred to as NeuronSeek) employs symbolic regression (SR) to discover the optimal neuron formulation and construct a network from these optimized neurons. Along this direction, this work replaces symbolic regression with tensor decomposition (TD) to discover optimal neuronal formulations, offering enhanced stability and faster convergence. Furthermore, we establish theoretical guarantees that modifying the aggregation functions with common activation functions can empower a network with a fixed number of parameters to approximate any continuous function with an arbitrarily small error, providing a rigorous mathematical foundation for the NeuronSeek framework. Extensive empirical evaluations demonstrate that our NeuronSeek-TD framework not only achieves superior stability, but also is competitive relative to the state-of-the-art models across diverse benchmarks. The code is available at https://github.com/HanyuPei22/NeuronSeek.

Figures (11)

• Figure 1: Task-driven neuron based on the vectorized symbolic regression (NS-SR).
• Figure 2: The overall framework of the proposed method. In the first stage, the input data, regardless of tables and images, are flattened into a vector representation and processed using an initial formula for neuronal search. A stable formula is then generated through CP decomposition. In the second stage, the neuronal formula is parameterized and integrated into various neural network backbones for task-specific applications.
• Figure 3: A rank-R CP decomposition of a third-order tensor.
• Figure 4: A sketch of our proof. In Steps 1 and 2, through interval partition, the approximation problem is transformed into a point-fitting problem. In Step 3, we use task-driven neurons to construct a unimodal function that can induce the dense trajectory. Then, composing the unimodal function can solve the point-fitting problem.
• Figure 5: A polynomial can be transformed into a unimodal map.

Theorems & Definitions (6)

• Theorem 1
• Proposition 1: katok1995introduction
• Definition 1: HUANG2005287
• Lemma 1: Dense Trajectory
• proof
• Proposition 2