BSAT: B-Spline Adaptive Tokenizer for Long-Term Time Series Forecasting

TL;DR

This work tackles the inefficiency and rigidity of patch-based transformers for long-horizon time-series forecasting by introducing BSAT, a parameter-free B-spline based tokenizer that adaptively allocates tokens to high-curvature regions, reducing self-attention complexity from $O(L^2)$ to $O(n^2)$ while handling irregular sampling. It couples BSAT with a Hybrid Additive Rotary Positional Encoding and a layer-wise learnable RoPE base (L-RoPE) to encode non-uniform tokens and enable multi-resolution attention across layers. Empirical results across ETTh1, Alabama PV, and ECL show BSAT achieving competitive RMSE, especially at low token budgets, and the ablations reveal that hybrid embeddings and diverse RoPE bases improve robustness and performance, though numerical stability can be an issue on highly volatile data. The combination of adaptive tokenization and flexible positional encoding offers practical efficiency gains for memory-constrained forecasting tasks, with avenues for further stability improvements and end-to-end differentiable tokenization.

Abstract

Long-term time series forecasting using transformers is hampered by the quadratic complexity of self-attention and the rigidity of uniform patching, which may be misaligned with the data's semantic structure. In this paper, we introduce the \textit{B-Spline Adaptive Tokenizer (BSAT)}, a novel, parameter-free method that adaptively segments a time series by fitting it with B-splines. BSAT algorithmically places tokens in high-curvature regions and represents each variable-length basis function as a fixed-size token, composed of its coefficient and position. Further, we propose a hybrid positional encoding that combines a additive learnable positional encoding with Rotary Positional Embedding featuring a layer-wise learnable base: L-RoPE. This allows each layer to attend to different temporal dependencies. Our experiments on several public benchmarks show that our model is competitive with strong performance at high compression rates. This makes it particularly well-suited for use cases with strong memory constraints.

Figures (7)

• Figure 1: Visualization of BSAT mapping 100 points into 12 tokens. Knot placement is densest in high-curvature segments. Top: Time series observations (black) and the resulting feature function (red). Middle: cumulative distribution function with quantiles and the resulting knot positions (blue triangles). Bottom: Observations, the final fitted B-spline curve, the basis functions that compose it, and the tokens that result from the basis functions. Each token is constructed of one coefficient $c_i$ and one center $\mu_i$.
• Figure 2: Model architecture: The network embeds coefficient and position channels, concatenates them, and applies both learnable positional encoding and RoPE.
• Figure 3: RMSE box-plot for token budget 45, showing the distribution of the top 15 runs for each configuration. RMSE is rescaled to $[0,1]$ inside each dataset $\times$ budget group.
• Figure 4: RoPE base layer convergence for one BSAT L-RoPE LPE run (16 dimensions, 4 heads) on Alabama PV. Layers specialize in different frequencies in an alternating pattern.
• Figure 5: Effect of learned RoPE base diversity. Cells show test RMSE change relative to the tightest-spread quartile (Q1) across quartiles Q2–Q4. Each cell shows the percent change in mean RMSE, negative values (blue) indicate performance improvements. The rightmost column shows the overall median spread per row. Wider spreads markedly help on Alabama PV and ECL and offer modest gains on ETTh1; the benefits diminishes as the token budget grows. Calculated on all runs.