Interpretable Probability Estimation with LLMs via Shapley Reconstruction

TL;DR

This work introduces PRISM (Probability Reconstruction via Shapley Measures), a framework for interpretable probability estimation with LLMs that decomposes an LLM's probability output into factor-wise Shapley contributions and reconstructs a calibrated final probability p_PRISM(x) = σ(φ_0 + ∑ φ_i). By obtaining φ_i via Comparative Prompting and averaging over random background sets, PRISM makes the prediction process transparent and capable of handling heterogeneous inputs, including unstructured text, while achieving improved predictive accuracy over direct prompting and several baselines across tabular and real-world domains. The Tabular-PRISM variant further enhances efficiency by batching background-set computations, and a factor-extraction pipeline ensures non-overlapping, complete feature sets for prediction. Extensive experiments on tabular datasets (Adult, Stroke, Heart Disease, Lending) and unstructured tasks (MIMIC-III readmission, agricultural prices, soccer) demonstrate PRISM’s robust performance, improved calibration, and provision of interpretable attributions that reveal how factors interact to shape probabilities in context. The work also discusses generalizations to multi-class tasks and interactions, along with reproducibility through an open repository.

Abstract

Large Language Models (LLMs) demonstrate potential to estimate the probability of uncertain events, by leveraging their extensive knowledge and reasoning capabilities. This ability can be applied to support intelligent decision-making across diverse fields, such as financial forecasting and preventive healthcare. However, directly prompting LLMs for probability estimation faces significant challenges: their outputs are often noisy, and the underlying predicting process is opaque. In this paper, we propose PRISM: Probability Reconstruction via Shapley Measures, a framework that brings transparency and precision to LLM-based probability estimation. PRISM decomposes an LLM's prediction by quantifying the marginal contribution of each input factor using Shapley values. These factor-level contributions are then aggregated to reconstruct a calibrated final estimate. In our experiments, we demonstrate PRISM improves predictive accuracy over direct prompting and other baselines, across multiple domains including finance, healthcare, and agriculture. Beyond performance, PRISM provides a transparent prediction pipeline: our case studies visualize how individual factors shape the final estimate, helping build trust in LLM-based decision support systems.

Figures (21)

• Figure 1: Illustration of direct LLM prompting and PRISM. PRISM first estimates Shapley values (factor contributions) and aggregates them to reconstruct final probability. We use red to represent the factors found by PRISM to have positive contribution to the positive outcome (have a stroke), green are factors found to be negative. The size reflects the contribution's absolute value. $f(\cdot)$ is from LLM prediction and $S$ is a background set, $\sigma(\cdot)$ is the sigmoid function (see details in Section \ref{['sec:method']}).
• Figure 2: Comparative Prompting.
• Figure 3: When calculating Shapley value of the factor "Age=79", we put multiple $S$ in one table. Factor values from reference instances are noted in green.
• Figure 4: PRISM attribution example from MIMIC-III (True Label=1), illustrating the contribution of each factor.
• Figure 5: PRISM-10_shot

Theorems & Definitions (5)

• Definition 1: Shapley value
• Definition 2
• Proposition 1
• Proposition 1
• proof