Gauging Overprecision in LLMs: An Empirical Study

TL;DR

The paper investigates overprecision in black-box LLMs by imposing explicit confidence levels and evaluating interval-based numerical answers through a three-phase framework of generation, refinement, and evaluation. It shows that LLMs are poorly calibrated for numerical tasks and that interval length does not reliably track the imposed confidence level, with precision depending on task and prompting. Refinement strategies yield limited gains, and self-refinement often degrades performance, challenging some cross-domain findings in cognitive science. These results provide baseline insights into overprecision in LLMs and highlight the need for robust, domain-aware evaluation and mitigation methods for confident but inaccurate numerical reasoning.

Abstract

Recently, overconfidence in large language models (LLMs) has garnered considerable attention due to its fundamental importance in quantifying the trustworthiness of LLM generation. However, existing approaches prompt the \textit{black box LLMs} to produce their confidence (\textit{verbalized confidence}), which can be subject to many biases and hallucinations. Inspired by a different aspect of overconfidence in cognitive science called \textit{overprecision}, we designed a framework for its study in black box LLMs. This framework contains three main phases: 1) generation, 2) refinement and 3) evaluation. In the generation phase we prompt the LLM to generate answers to numerical questions in the form of intervals with a certain level of confidence. This confidence level is imposed in the prompt and not required for the LLM to generate as in previous approaches. We use various prompting techniques and use the same prompt multiple times to gauge the effects of randomness in the generation process. In the refinement phase, answers from the previous phase are refined to generate better answers. The LLM answers are evaluated and studied in the evaluation phase to understand its internal workings. This study allowed us to gain various insights into LLM overprecision: 1) LLMs are highly uncalibrated for numerical tasks 2) there is no correlation between the length of the interval and the imposed confidence level, which can be symptomatic of a a) lack of understanding of the concept of confidence or b) inability to adjust self-confidence by following instructions, {3) LLM numerical precision differs depending on the task, scale of answer and prompting technique 4) Refinement of answers doesn't improve precision in most cases. We believe this study offers new perspectives on LLM overconfidence and serves as a strong baseline for overprecision in LLMs.

Figures (5)

• Figure 1: An outline of the precision elicitation framework and an example. Given an input question, a confidence level is first specified, a prompt strategy is then chosen, and the confidence level is integrated into the prompt. Next, the sampling strategy and the number of samples are determined to control the amount and diversity of outputs of the same prompt. After that, an aggregator combines the different answers to produce the most likely answer.
• Figure 2: Scale affect on precision: These figures show the distribution of the hit average for different answers in the vanilla prompt setting for different models on different datasets. The figures demonstrate that the performance Is affected by the prompting strategy, the scale of the answer, and the task.
• Figure 3: The hit average metric as a function of the number of examples provided in the self-refinement prompt. The titles of the subfigures are organized as follows: [setting][dataset][kind]. The setting can either be Single or mixed (refer to the experimental protocol for more detail). The kind can either be "chosen" for answers that were selected by the LLM to be the most correct. The kind can also be "proposed" for the answers that were proposed by the LLM but didn't exist in the provided examples.
• Figure 4: The figures show the distribution of the average DS metric across confidence levels for different datasets, in different models for vanilla and CoT prompts. GPT-3.5 is short for GPT-3.5-turbo, and GPT-4o is short for GPT-4o-mini. The DS values are lowest for MMLU, higher for the Medical dataset, and highest for FinQA. This supports earlier results in Tables \ref{['tab:overprec']} and \ref{['tab:refine_agg_res_single']} and Section \ref{['sec:res_eval']}, confirming that the observed trends are consistent across datasets and not driven by outliers.
• Figure 5: The figures show the distribution of the average ILS metric across confidence levels for different datasets, in different models for vanilla and CoT prompts. GPT-3.5 is short for GPT-3.5-turbo, and GPT-4o is short for GPT-4o-mini. The figures show that interval lengths are largest in FinQA, followed by Medical, and smallest in MMLU. This suggests that LLMs adjust interval size based on task difficulty, reflecting an awareness of uncertainty. However, combined with earlier findings on the lack of correlation between confidence and interval size, it indicates that while LLMs sense task hardness, they struggle to align their confidence with explicit instructions.