Paper
Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
Authors
Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu
Abstract
Tabular data remain the predominant format for real-world applications. Yet,
developing effective neural models for tabular data remains challenging due to
heterogeneous feature types and complex interactions occurring at multiple
scales. Recent advances in tabular in-context learning (ICL), such as TabPFN
and TabICL, have achieved state-of-the-art performance comparable to
gradient-boosted trees (GBTs) without task-specific fine-tuning. However,
current architectures exhibit key limitations: (1) single-scale feature
processing that overlooks hierarchical dependencies, (2) dense attention with
quadratic scaling in table width, and (3) strictly sequential component
processing that prevents iterative representation refinement and
cross-component communication. To address these challenges, we introduce
Orion-MSP, a tabular ICL architecture featuring three key innovations: (1)
multi-scale processing to capture hierarchical feature interactions; (2)
block-sparse attention combining windowed, global, and random patterns for
scalable efficiency and long-range connectivity; and (3) a Perceiver-style
memory enabling safe bidirectional information flow across components. Across
diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance
while scaling effectively to high-dimensional tables, establishing a new
standard for efficient tabular in-context learning. The model is publicly
available at https://github.com/Lexsi-Labs/Orion-MSP .