Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

Yael S. Elmatad

TL;DR

This tokenizer uses a deliberately simplistic discretized vocabulary so it can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing.

Abstract

Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.

Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting

TL;DR

This tokenizer uses a deliberately simplistic discretized vocabulary so it can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing.

Abstract

Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.
Paper Structure (33 sections, 4 equations, 8 figures, 4 tables)

This paper contains 33 sections, 4 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tabular Transformers
  4. Ordinal Regression and Calibration
  5. Event Trajectories and Temporal Point Processes
  6. Discretization for Forecasting
  7. Method
  8. Problem Setup
  9. Discretization Strategy
  10. Gaussian-integrated Soft Targets
  11. Adaptive Smoothing
  12. Architecture
  13. Experiments
  14. Setup
  15. Results
  16. ...and 18 more sections

Figures (8)

  • Figure 1: Top: MAE vs. history length. Middle: normalized MAE showing the ablation gap relative to the full model. Bottom: MAE vs. week gap between the penultimate and target race, highlighting temporal staleness.
  • Figure 2: Q–Q plot showing distributions across various percentiles from $\sigma=0$ (cross entropy) to $\sigma=35$ (wide Gaussian).
  • Figure 3: Quantile calibration emphasizing where predicted densities concentrate. Note that small $\sigma$ values concentrate predictions at the extremes, suggesting model overconfidence, whereas large $\sigma$ values appear underconfident, with larger probability mass in the middle.
  • Figure 4: Calibration plots highlighting how different smoothing choices affect residual drift across percentiles.
  • Figure 5: YE (short-career) runner trajectory with predicted PDFs illustrating tight uncertainty as the history grows.
  • ...and 3 more figures