Embedding Perturbation may Better Reflect the Uncertainty in LLM Reasoning
Qihao Wen, Jiahao Wang, Yang Nan, Pengfei He, Ravi Tandon, Han Xu
TL;DR
This work addresses the problem of estimating uncertainty in intermediate steps of LLM reasoning, not just the final answer. It introduces perturbation-based token-level UQ metrics that measure how sensitive a token's generation is to small perturbations in preceding token embeddings, using random Gaussian perturbations and gradient-based adversarial perturbations to compute a token-level uncertainty score $\mathrm{Pert.}(x_t)$. Across pure logic and math reasoning tasks, these perturbation-based metrics outperform traditional signals like token probability and entropy in locating uncertain steps and are substantially more efficient than multiple-sampling approaches. While they offer clear benefits for intermediate reasoning auditing, they show limited effectiveness for factual hallucination detection and may reflect pathway variability rather than correctness alone, indicating room for integration with additional uncertainty signals for safer, more reliable LLM reasoning.
Abstract
Large language Models (LLMs) have achieved significant breakthroughs across diverse domains; however, they can still produce unreliable or misleading outputs. For responsible LLM application, Uncertainty Quantification (UQ) techniques are used to estimate a model's uncertainty about its outputs, indicating the likelihood that those outputs may be problematic. For LLM reasoning tasks, it is essential to estimate the uncertainty not only for the final answer, but also for the intermediate steps of the reasoning, as this can enable more fine-grained and targeted interventions. In this study, we explore what UQ metrics better reflect the LLM's ``intermediate uncertainty''during reasoning. Our study reveals that an LLMs' incorrect reasoning steps tend to contain tokens which are highly sensitive to the perturbations on the preceding token embeddings. In this way, incorrect (uncertain) intermediate steps can be readily identified using this sensitivity score as guidance in practice. In our experiments, we show such perturbation-based metric achieves stronger uncertainty quantification performance compared with baseline methods such as token (generation) probability and token entropy. Besides, different from approaches that rely on multiple sampling, the perturbation-based metrics offer better simplicity and efficiency.