Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SG-XDEAT: Sparsity-Guided Cross-Dimensional and Cross-Encoding Attention with Target-Aware Conditioning in Tabular Learning

Chih-Chuan Cheng, Yi-Ju Tseng

TL;DR

SG-XDEAT addresses tabular learning by jointly modeling raw features and target-aware encodings through a dual-stream Transformer. It introduces Cross-Encoding Self-Attention and Cross-Dimension Self-Attention, augmented with Adaptive Sparse Self-Attention to suppress noisy signals. Empirical results across five benchmarks show competitive or state-of-the-art performance, with ablations confirming the benefits of combining cross-view and cross-feature interactions and of noise suppression. The approach narrows the gap between deep tabular models and gradient-boosted trees and offers robust, label-informed representations for tabular prediction tasks.

Abstract

We propose SG-XDEAT (Sparsity-Guided Cross Dimensional and Cross-Encoding Attention with Target Aware Conditioning), a novel framework designed for supervised learning on tabular data. At its core, SG-XDEAT employs a dual-stream encoder that decomposes each input feature into two parallel representations: a raw value stream and a target-conditioned (label-aware) stream. These dual representations are then propagated through a hierarchical stack of attention-based modules. SG-XDEAT integrates three key components: (i) Cross-Dimensional self-attention, which captures intra-view dependencies among features within each stream; (ii) Cross-Encoding self-attention, which enables bidirectional interaction between raw and target-aware representations; and (iii) an Adaptive Sparse Self-Attention (ASSA) mechanism, which dynamically suppresses low-utility tokens by driving their attention weights toward zero--thereby mitigating the impact of noise. Empirical results on multiple public benchmarks show consistent gains over strong baselines, confirming that jointly modeling raw and target-aware views--while adaptively filtering noise--yields a more robust deep tabular learner.

SG-XDEAT: Sparsity-Guided Cross-Dimensional and Cross-Encoding Attention with Target-Aware Conditioning in Tabular Learning

TL;DR

SG-XDEAT addresses tabular learning by jointly modeling raw features and target-aware encodings through a dual-stream Transformer. It introduces Cross-Encoding Self-Attention and Cross-Dimension Self-Attention, augmented with Adaptive Sparse Self-Attention to suppress noisy signals. Empirical results across five benchmarks show competitive or state-of-the-art performance, with ablations confirming the benefits of combining cross-view and cross-feature interactions and of noise suppression. The approach narrows the gap between deep tabular models and gradient-boosted trees and offers robust, label-informed representations for tabular prediction tasks.

Abstract

We propose SG-XDEAT (Sparsity-Guided Cross Dimensional and Cross-Encoding Attention with Target Aware Conditioning), a novel framework designed for supervised learning on tabular data. At its core, SG-XDEAT employs a dual-stream encoder that decomposes each input feature into two parallel representations: a raw value stream and a target-conditioned (label-aware) stream. These dual representations are then propagated through a hierarchical stack of attention-based modules. SG-XDEAT integrates three key components: (i) Cross-Dimensional self-attention, which captures intra-view dependencies among features within each stream; (ii) Cross-Encoding self-attention, which enables bidirectional interaction between raw and target-aware representations; and (iii) an Adaptive Sparse Self-Attention (ASSA) mechanism, which dynamically suppresses low-utility tokens by driving their attention weights toward zero--thereby mitigating the impact of noise. Empirical results on multiple public benchmarks show consistent gains over strong baselines, confirming that jointly modeling raw and target-aware views--while adaptively filtering noise--yields a more robust deep tabular learner.
Paper Structure (39 sections, 12 equations, 3 figures, 16 tables)

This paper contains 39 sections, 12 equations, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dual-Stream Embedding
  5. Token Extension
  6. Dual-Path Transformer
  7. Cross-Encoding Self-Attention
  8. Cross-Dimension Self-Attention
  9. Token Aggregation
  10. Classifier / Regressor
  11. Experiments
  12. Experimental Setup
  13. Dataset
  14. Baseline
  15. Implementation Details
  16. ...and 24 more sections

Figures (3)

  • Figure 1: The overview of SG-XDEAT. L is the layer number.
  • Figure 2: Illustration of architectural variants used in the ablation study.
  • Figure 3: Regression performance (RMSE) on synthetic datasets with and without ASSA. A lower $\rho$ implies a higher proportion of irrelevant (noisy) features. Lower RMSE indicates better predictive performance.