Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors

TL;DR

The paper investigates why transformers underperform on long-range sequence benchmarks when trained from scratch and demonstrates that self-pretraining (SPT) on downstream data provides strong, data-driven priors. By applying SPT across multiple architectures and tasks, the authors show that vanilla Transformers can match state-space models (SSMs) like S4 on Long Range Arena (LRA) and achieve a 20-point absolute improvement on PathX-256, substantially narrowing prior gaps. They further reveal that handcrafted priors offer limited advantage once task-specific priors are learned via denoising objectives, and that SPT is especially beneficial with limited data. The work advocates incorporating data-driven pretraining as a standard step in architecture evaluation and provides extensive analyses across modalities, data scales, and kernels, underscoring practical gains and broader applicability.

Abstract

Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences. However, these impressive empirical gains have been by and large demonstrated on benchmarks (e.g. Long Range Arena), where models are randomly initialized and trained to predict a target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures and that pretraining with standard denoising objectives, using $\textit{only the downstream task data}$, leads to dramatic gains across multiple architectures and to very small gaps between Transformers and state space models (SSMs). In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points. Subsequently, we analyze the utility of previously-proposed structured parameterizations for SSMs and show they become mostly redundant in the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.

Figures (5)

• Figure 1: Evaluation of Transformers and S4 on Long Range Arena when trained from scratch vs. when self pretrained.
• Figure 2: Average performance of models when trained from scratch or self pretrained, for different sets of initializations prior to pretraining. See Table \ref{['init-role-state-spaces-table']} for per-task results.
• Figure 3: Trained from scratch and self pretrained (SPT) versions of S4 evaluated on multiple data scales for Image and Text tasks from LRA, originally containing $45K$ and $25K$ samples respectively. (left) absolute performances and (right) relative gains due to SPT over training from scratch.
• Figure 4: Maximal absolute values of kernels across channels in S4 learned via self pretraining (PT) compared against the standard HiPPO kernels (Baseline). Only odd layers are shown for better visualization.
• Figure 5: Training accuracy on the downstream and denoising task of Transformers on Image and Text from LRA, across epochs, showing the denoising task is solved relatively quickly (orange) and that finetuned models optimize faster (solid vs dashed blue curves). Training budget is set to 30 epochs for Text and 150 epochs for Image.