Advancing Sequential Numerical Prediction in Autoregressive Models

TL;DR

This work tackles the mismatch between standard cross-entropy training and the ordinal, holistic nature of numerical sequences in autoregressive models. It introduces Numerical Token Integrity Loss (NTIL), a two-level loss combining token-level Earth Mover's Distance with exponential position weighting and a sequence-level multi-token optimization via differentiable numeric reconstruction using Gumbel-softmax. The approach is validated across diverse vision-language and numeric-reasoning tasks, showing consistent gains over CE and traditional EMD on both MLLMs and LLMs, though some models exhibit varying magnitudes of improvement. NTIL demonstrates practical value for precise numerical outputs in multimodal and textual reasoning settings, while also outlining limitations and avenues for adaptive weighting and efficiency improvements.

Abstract

Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.

Figures (6)

• Figure 1: Sequence-level digit token loss illustration.
• Figure 2: Cross-entropy fails to distinguish predictions, whereas EMD correlates smaller loss for better predicted distributions.
• Figure 3: Constructing a numerical value from tokens.
• Figure 4: Results for quantitative analysis.
• Figure 5: The illustrations for each task.