Sprecher Networks: A Parameter-Efficient Kolmogorov-Arnold Architecture

TL;DR

Sprecher Networks propose a deep architecture built from blocks that realize Sprecher's 1965 shift-and-sum formula using shared monotone and general splines, with learnable shifts and vector mixing to enable parameter-efficient function approximation. The design achieves linear in width parameter scaling and memory-efficient forward computation via sequential evaluation, differentiating it from MLPs and edge-activation networks like KANs. Empirical results on synthetic regression and PDE benchmarks show SNs can match or outperform KANs at matched budgets, with lateral mixing especially beneficial for vector-valued outputs. The work also provides a detailed implementation blueprint, including domain propagation bounds and memory-saving strategies, and discusses theoretical open questions around deep universality and the role of lateral mixing. Overall, SNs offer a compelling, theory-grounded approach to building efficient, interpretable function-approximators with strong potential for scientific and memory-constrained applications.

Abstract

We present Sprecher Networks (SNs), a family of trainable neural architectures inspired by the classical Kolmogorov-Arnold-Sprecher (KAS) construction for approximating multivariate continuous functions. Distinct from Multi-Layer Perceptrons (MLPs) with fixed node activations and Kolmogorov-Arnold Networks (KANs) featuring learnable edge activations, SNs utilize shared, learnable splines (monotonic and general) within structured blocks incorporating explicit shift parameters and mixing weights. Our approach directly realizes Sprecher's specific 1965 sum of shifted splines formula in its single-layer variant and extends it to deeper, multi-layer compositions. We further enhance the architecture with optional lateral mixing connections that enable intra-block communication between output dimensions, providing a parameter-efficient alternative to full attention mechanisms. Beyond parameter efficiency with $O(LN + LG)$ scaling (where $G$ is the knot count of the shared splines) versus MLPs' $O(LN^2)$, SNs admit a sequential evaluation strategy that reduces peak forward-intermediate memory from $O(N^2)$ to $O(N)$ (treating batch size as constant), making much wider architectures feasible under memory constraints. We demonstrate empirically that composing these blocks into deep networks leads to highly parameter and memory-efficient models, discuss theoretical motivations, and compare SNs with related architectures (MLPs, KANs, and networks with learnable node activations).

Figures (10)

• Figure 1: Data flow through a single Sprecher block. Each input $x_i$ is shifted by $\eta q$ (where $q$ indexes outputs), passed through the shared monotonic spline $\phi$, weighted by $\lambda_i$, and summed. The pre-activation $s_q = \sum_i \lambda_i \phi(x_i + \eta q) + \alpha q$ undergoes optional lateral mixing before being transformed by the shared general spline $\Phi$. Residual connections (dashed) provide direct gradient paths.
• Figure 2: Internal structure of a Sprecher block showing spline sharing. Unlike KANs where each edge has a unique learnable spline, a Sprecher block uses only two shared splines: one monotonic $\phi$ applied to all shifted inputs, and one general $\Phi$ applied to all weighted sums. The mixing weights $\lambda_i$ are shared across all output dimensions. Diversity across outputs arises from the index-dependent shifts $+\eta q$ (applied before $\phi$) and $+\alpha q$ (added to each sum). This extreme parameter sharing yields $O(d_{\mathrm{in}} + G)$ parameters per block versus $O(d_{\mathrm{in}} \cdot d_{\mathrm{out}} \cdot G)$ for KANs.
• Figure 3: Lateral mixing enables cross-dimensional communication before the outer spline $\Phi$. In the cyclic variant (shown), each pre-activation $s_q$ receives a scaled contribution from its neighbor $s_{(q+1) \bmod d_{\mathrm{out}}}$, parameterized by scale $\tau$ and per-output weights $\omega_q$. This adds only $O(d_{\mathrm{out}})$ parameters while breaking the symmetries inherent in the shared-weight structure. The bidirectional variant additionally includes contributions from $s_{(q-1) \bmod d_{\mathrm{out}}}$.
• Figure 4: Dimension-adaptive cyclic residual connections (indices shown 1-based for readability). Left: When dimensions match, a single scalar weight applies element-wise. Center: When $d_{\mathrm{in}} > d_{\mathrm{out}}$ (pooling), multiple inputs are cyclically assigned to each output via modular indexing. Right: When $d_{\mathrm{in}} < d_{\mathrm{out}}$ (broadcast), inputs are cyclically reused across outputs. All cases maintain $O(\max(d_{\mathrm{in}}, d_{\mathrm{out}}))$ parameters, preserving the linear scaling of the architecture.
• Figure 5: The dual spline system in Sprecher Networks. Left: The inner spline $\phi$ is monotonic (non-decreasing) with fixed codomain $[0, 1]$, parameterized via cumulative sums of softplus-transformed increments; it is strictly increasing on its spline domain and uses constant extension outside that domain ($0$ for inputs below the leftmost knot, $1$ above the rightmost knot). Right: The outer spline $\Phi$ is a general (non-monotonic) univariate spline (piecewise-linear or cubic PCHIP) with optional learnable codomain parameters $(c_c, c_r)$ defining center and radius. It uses linear extrapolation outside its domain. Both domains can be updated during training as $\lambda$ and $\eta$ evolve; in some experiments we freeze domain updates after a warm-up period (and in our PINN experiments we disable domain updates altogether).

Theorems & Definitions (35)

• Definition 1: Network notation
• Remark 1: Computational considerations
• Remark 2: Lateral mixing as structured attention
• Remark 3: Lateral mixing resolves the shared-weight optimization plateau
• Remark 4: Cyclic residuals as dimensional folding
• Remark 5: Structured sparsity and architectural coherence
• Remark 6: On the nature of composition
• Remark 7: Lateral mixing in expanded form
• Remark 8
• Remark 9: Necessity of internal shifts