Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
Tabular data remain the predominant format for real-world applications. Yet,
developing effective neural models for tabular data remains challenging due to
heterogeneous feature types and complex interactions occurring at multiple
scales. Recent advances in tabular in-context learning (ICL), such as TabPFN
and TabICL, have achieved state-of-the-art performance comparable to
gradient-boosted trees (GBTs) without task-specific fine-tuning. However,
current architectures exhibit key limitations: (1) single-scale feature
processing that overlooks hierarchical dependencies, (2) dense attention with
quadratic scaling in table width, and (3) strictly sequential component
processing that prevents iterative representation refinement and
cross-component communication. To address these challenges, we introduce
Orion-MSP, a tabular ICL architecture featuring three key innovations: (1)
multi-scale processing to capture hierarchical feature interactions; (2)
block-sparse attention combining windowed, global, and random patterns for
scalable efficiency and long-range connectivity; and (3) a Perceiver-style
memory enabling safe bidirectional information flow across components. Across
diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance
while scaling effectively to high-dimensional tables, establishing a new
standard for efficient tabular in-context learning. The model is publicly
available at https://github.com/Lexsi-Labs/Orion-MSP .
690b348b60494e4fa76754f6
Mohamed Bouadi
690b348b60494e4fa76754f7
Pratinav Seth
claimed_verified
2025-11-05T16:24:08.560Z
66fce04d927ec45504514afd
https://cdn-avatars.huggingface.co/v1/production/uploads/66fce04d927ec45504514afd/lA_bTErY7JywT6xbzdflo.jpeg
Pratinav Seth
pratinavsetharya
user
pratinavsetharya
690b348b60494e4fa76754f8
Aditya Tanna
690b348b60494e4fa76754f9
Vinay Kumar Sankarapu
Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
Tabular data remain the predominant format for real-world applications. Yet,
developing effective neural models for tabular data remains challenging due to
heterogeneous feature types and complex interactions occurring at multiple
scales. Recent advances in tabular in-context learning (ICL), such as TabPFN
and TabICL, have achieved state-of-the-art performance comparable to
gradient-boosted trees (GBTs) without task-specific fine-tuning. However,
current architectures exhibit key limitations: (1) single-scale feature
processing that overlooks hierarchical dependencies, (2) dense attention with
quadratic scaling in table width, and (3) strictly sequential component
processing that prevents iterative representation refinement and
cross-component communication. To address these challenges, we introduce
Orion-MSP, a tabular ICL architecture featuring three key innovations: (1)
multi-scale processing to capture hierarchical feature interactions; (2)
block-sparse attention combining windowed, global, and random patterns for
scalable efficiency and long-range connectivity; and (3) a Perceiver-style
memory enabling safe bidirectional information flow across components. Across
diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance
while scaling effectively to high-dimensional tables, establishing a new
standard for efficient tabular in-context learning. The model is publicly
available at https://github.com/Lexsi-Labs/Orion-MSP .
Orion-MSP, a tabular in-context learning architecture, addresses limitations in current models by incorporating multi-scale processing, block-sparse attention, and a Perceiver-style memory, achieving state-of-the-art performance on diverse benchmarks.
tabular in-context learning TabPFN TabICL gradient-boosted trees multi-scale processing block-sparse attention Perceiver-style memory