NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction
Peter Røysland Aarnes, Vinay Setty
TL;DR
NumPert introduces a perturbation-based benchmark to stress-test veracity prediction in large language models on numerical claims. By perturbing numerical values with six probe types and evaluating zero-shot, two-shot, and perturbation-aware prompting across diverse models, the study reveals widespread vulnerability and the absence of a universally robust model. Key insights show that masking and neg-number perturbations are particularly challenging, longer context and reasoning chains correlate with errors, and perturbation-aware prompting can partially recover performance for reasoning-enabled systems. The work highlights critical gaps in numerical fact-checking and offers a principled framework and dataset to drive improvements in robustness and trustworthy numerical reasoning in long-context settings.
Abstract
Large language models show strong performance on knowledge intensive tasks such as fact-checking and question answering, yet they often struggle with numerical reasoning. We present a systematic evaluation of state-of-the-art models for veracity prediction on numerical claims and evidence pairs using controlled perturbations, including label-flipping probes, to test robustness. Our results indicate that even leading proprietary systems experience accuracy drops of up to 62\% under certain perturbations. No model proves to be robust across all conditions. We further find that increasing context length generally reduces accuracy, but when extended context is enriched with perturbed demonstrations, most models substantially recover. These findings highlight critical limitations in numerical fact-checking and suggest that robustness remains an open challenge for current language models.