Agile Reinforcement Learning through Separable Neural Architecture

TL;DR

This work introduces SPAN, a spline-based, separable neural architecture for reinforcement learning that is tailored for resource-constrained settings. By integrating a learnable preprocessing layer with a rank-$M$ tensor product of B-spline bases, SPAN provides intrinsic parameter and sample efficiency, outperforming MLP baselines across online and offline benchmarks and delivering superior anytime performance. The approach achieves 30–50% improvements in sample efficiency and 1.3–9× higher success rates, with robust performance across Classic Control, MuJoCo, and Adroit offline tasks, suggesting strong practical impact for robotics, edge devices, and production systems. SPAN demonstrates that structured function approximators with local smoothness biases can significantly enhance learning efficiency where data, memory, or training budgets are limited.

Abstract

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet the go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Recent spline-based separable architectures - such as Kolmogorov-Arnold Networks (KANs) - have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks), a novel function approximation approach to RL. SPAN adapts the low rank KHRONOS framework by integrating a learnable preprocessing layer with a separable tensor product B-spline basis. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, as well as offline settings (Minari/D4RL). Empirical results demonstrate that SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high performance alternative for learning intrinsically efficient policies in resource-limited settings.

Figures (9)

• Figure 1: SPAN architecture for actor-critic reinforcement learning. Environment states, action or state-action pair pass through a preprocessing layer (Dense + Sigmoid) that normalizes inputs to $[0,1]^d$. Per-dimension B-spline basis expansions generate univariate features, which are combined via tensor products (outer product) across dimensions to form $M$ separable modes. A linear head produces task specific outputs: policy logits (with softmax) for discrete actions or Gaussian parameters (mean, std) for continuous actions. Both actor and critic use identical SPAN architectures with separate parameters.
• Figure 2: Reward curves for Classic Control and Box2D environments showing mean performance ± 1 standard deviation across 20 seeds.
• Figure 3: Learning curves for MuJoCo continuous control environments showing mean episodic return $\pm$ one standard deviation across 20 random seeds.
• Figure 4: Ablation study on Acrobot-v1 showing performance vs. nmode, nelem, and degree.
• Figure 5: Ablation study on HalfCheetah-v5 showing performance vs. krank, kelem.