Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Optimization of Neural Autoregressors via Scoring rules

Jonas Landsgesell

TL;DR

The paper tackles the problem of extending non-parametric distributional regression to multivariate outputs without the prohibitive memory and data requirements of explicit joint grids. It introduces JonasNet, an autoregressive neural architecture (RNN or causally masked Transformer) that models $P(\mathbf{y}|\mathbf{x})$ as a sequence of discretized conditional densities and trains it with strictly proper scoring rules (Energy Score and variogram-based scores) to ensure sharpness and calibration. Key contributions include a low-rank independent-parameterization option, direct PMF sampling with de-quantization, and empirical evidence showing improved multivariate forecasting and uncertainty quantification over univariate baselines. The approach promises scalable, non-parametric multivariate density estimation suitable for integration into Tabular Foundation Models, enabling robust uncertainty-aware predictions in high-dimensional settings.

Abstract

Non-parametric distributional regression has achieved significant milestones in recent years. Among these, the Tabular Prior-Data Fitted Network (TabPFN) has demonstrated state-of-the-art performance on various benchmarks. However, a challenge remains in extending these grid-based approaches to a truly multivariate setting. In a naive non-parametric discretization with $N$ bins per dimension, the complexity of an explicit joint grid scales exponentially and the paramer count of the neural networks rise sharply. This scaling is particularly detrimental in low-data regimes, as the final projection layer would require many parameters, leading to severe overfitting and intractability.

Joint Optimization of Neural Autoregressors via Scoring rules

TL;DR

The paper tackles the problem of extending non-parametric distributional regression to multivariate outputs without the prohibitive memory and data requirements of explicit joint grids. It introduces JonasNet, an autoregressive neural architecture (RNN or causally masked Transformer) that models as a sequence of discretized conditional densities and trains it with strictly proper scoring rules (Energy Score and variogram-based scores) to ensure sharpness and calibration. Key contributions include a low-rank independent-parameterization option, direct PMF sampling with de-quantization, and empirical evidence showing improved multivariate forecasting and uncertainty quantification over univariate baselines. The approach promises scalable, non-parametric multivariate density estimation suitable for integration into Tabular Foundation Models, enabling robust uncertainty-aware predictions in high-dimensional settings.

Abstract

Non-parametric distributional regression has achieved significant milestones in recent years. Among these, the Tabular Prior-Data Fitted Network (TabPFN) has demonstrated state-of-the-art performance on various benchmarks. However, a challenge remains in extending these grid-based approaches to a truly multivariate setting. In a naive non-parametric discretization with bins per dimension, the complexity of an explicit joint grid scales exponentially and the paramer count of the neural networks rise sharply. This scaling is particularly detrimental in low-data regimes, as the final projection layer would require many parameters, leading to severe overfitting and intractability.
Paper Structure (15 sections, 9 equations, 2 figures, 1 table)

This paper contains 15 sections, 9 equations, 2 figures, 1 table.

Table of Contents

  1. Abstract
  2. Introduction
  3. The Curse of Dimensionality in Naive Multivariate Approaches
  4. Proposed Solution
  5. Methodology
  6. Scoring Rules and Probabilistic Forecasting
  7. Discrete Energy Score on a Grid
  8. Overcoming the Dimensionality Constraint
  9. Results
  10. Inference
  11. Quantifying Uncertainty
  12. Drawing samples from the estimated PMF
  13. Contribution
  14. Outlook
  15. Appendix

Figures (2)

  • Figure 1: Architecture of JonasNet.
  • Figure 2: Posterior Predictive Density at $x=1.29$. The blue contours represent the JonasNet predicted distribution $P(\mathbf{y}|x)$, capturing the heteroscedastic coupling. While the actual noisy observation (green circle) is displaced by stochastic fluctuations, the JonasNet mean (red X) provides a superior approximation of the ideal ground truth (magenta star) compared to the XGBoost baseline with two independently fitted models per dimension(orange square).