Uncertainty Quantification and Decomposition for LLM-based Recommendation

TL;DR

This work tackles the reliability challenge of LLM-based recommendations by introducing a framework to quantify predictive uncertainty as the entropy of the predictive ranking distribution over candidates. It further decomposes this uncertainty into prompt uncertainty and recommendation uncertainty using a latent prompting variable Pu and a Plackett-Luce based approximation, enabling diagnostics of volatility sources. The approach is validated on real-world datasets with multiple LLMs, showing that predictive uncertainty correlates with recommendation quality and that uncertainty-aware prompting can improve performance with minimal changes to prompt length. The study provides practical guidance for deploying LLMs in recommendation settings and contributes a publicly available implementation to support trustworthy use of these models.

Abstract

Despite the widespread adoption of large language models (LLMs) for recommendation, we demonstrate that LLMs often exhibit uncertainty in their recommendations. To ensure the trustworthy use of LLMs in generating recommendations, we emphasize the importance of assessing the reliability of recommendations generated by LLMs. We start by introducing a novel framework for estimating the predictive uncertainty to quantitatively measure the reliability of LLM-based recommendations. We further propose to decompose the predictive uncertainty into recommendation uncertainty and prompt uncertainty, enabling in-depth analyses of the primary source of uncertainty. Through extensive experiments, we (1) demonstrate predictive uncertainty effectively indicates the reliability of LLM-based recommendations, (2) investigate the origins of uncertainty with decomposed uncertainty measures, and (3) propose uncertainty-aware prompting for a lower predictive uncertainty and enhanced recommendation. Our source code and model weights are available at https://github.com/WonbinKweon/UNC_LLM_REC_WWW2025

Figures (9)

• Figure 1: Example prompt for list-wise ranking with LLMs.
• Figure 2: Violin plots describing the user distributions with the coefficient of variation.
• Figure 3: Generation volatility and recommendation performance of fine-tuned LLMs. We plot the rank variance of candidate items for each user group. G1 has the highest average N@20 and G4 has the lowest average N@20.
• Figure 4: Analysis of fine-tuned ranking on MovieLens 1M. %Rec. Unc. and %Pro. Unc. represent the proportion of recommendation and prompt uncertainty within the total uncertainty, respectively. For (d), the leave-one-out ground-truth item is not guaranteed to be included in the candidate set.
• Figure : (a) "Candidate [A]"$\rightarrow$"Movie [A]"