TFWT: Tabular Feature Weighting with Transformer

TL;DR

TFWT tackles the challenge of nonuniform feature importance in tabular data by introducing a Transformer-based feature weighting framework that learns context-aware feature weights. It integrates discrete-continuous feature alignment, multi-head self-attention for weighting, and a cross-attention decoder to produce a weighted feature matrix $\mathbf{F_{rew}} = \mathbf{W} \odot \mathbf{F}$, followed by a PPO-based fine-tuning loop to minimize information redundancy. The approach is validated on four real-world datasets across multiple downstream tasks, showing substantial accuracy and AUC gains and reduced variance with fine-tuning, often outperforming the TabTransformer baseline. The work demonstrates the practical impact of combining Transformer-based feature modeling with reinforcement learning for robust tabular data analysis.

Abstract

In this paper, we propose a novel feature weighting method to address the limitation of existing feature processing methods for tabular data. Typically the existing methods assume equal importance across all samples and features in one dataset. This simplified processing methods overlook the unique contributions of each feature, and thus may miss important feature information. As a result, it leads to suboptimal performance in complex datasets with rich features. To address this problem, we introduce Tabular Feature Weighting with Transformer, a novel feature weighting approach for tabular data. Our method adopts Transformer to capture complex feature dependencies and contextually assign appropriate weights to discrete and continuous features. Besides, we employ a reinforcement learning strategy to further fine-tune the weighting process. Our extensive experimental results across various real-world datasets and diverse downstream tasks show the effectiveness of TFWT and highlight the potential for enhancing feature weighting in tabular data analysis.

Figures (8)

• Figure 1: A Demonstration of feature weighting. Traditional feature weighting methods assign the same weight to one feature. Our weighting method assigns different weights to different samples in one feature.
• Figure 2: The framework consists of three components. In the alignment we convert discrete ($f_1$ to $f_M$) and continuous ($f_{M+1}$ to $f_K$) features into uniform-length vectors. In the weighting we initialize and reassign weights according to feature relationships. The fine-tuning process employs reinforcement learning to refine the weighting model.
• Figure 3: Accuracy Improvement Comparison.
• Figure 3: Comparison of Mean AUC.
• Figure 4: Comparison on MLP (Accuracy and F1).