Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems

Chandan Kumar Sah, Xiaoli Lian, Li Zhang, Tony Xu, Syed Shazaib Shah

TL;DR

The paper addresses the dual challenges of predictive uncertainty and fairness in LLM-based recommendations (RecLLMs). It proposes an uncertainty-aware evaluation framework and a benchmark dataset annotated for eight demographic attributes across movies and music, enabling multi-faceted bias and reliability analyses. Using Google Gemini 1.5 Flash, it uncovers systematic fairness gaps (e.g., SNSR = $0.1363$, SNSV = $0.0507$) that persist under prompt perturbations, and introduces personality-aware fairness metrics to reveal bias linked to user traits. The work lays a foundation for safer, more interpretable RecLLMs by integrating uncertainty quantification with fairness, and outlines directions toward multi-model benchmarks and adaptive calibration for trustworthy deployment.

Abstract

Large language models (LLMs) enable powerful zero-shot recommendations by leveraging broad contextual knowledge, yet predictive uncertainty and embedded biases threaten reliability and fairness. This paper studies how uncertainty and fairness evaluations affect the accuracy, consistency, and trustworthiness of LLM-generated recommendations. We introduce a benchmark of curated metrics and a dataset annotated for eight demographic attributes (31 categorical values) across two domains: movies and music. Through in-depth case studies, we quantify predictive uncertainty (via entropy) and demonstrate that Google DeepMind's Gemini 1.5 Flash exhibits systematic unfairness for certain sensitive attributes; measured similarity-based gaps are SNSR at 0.1363 and SNSV at 0.0507. These disparities persist under prompt perturbations such as typographical errors and multilingual inputs. We further integrate personality-aware fairness into the RecLLM evaluation pipeline to reveal personality-linked bias patterns and expose trade-offs between personalization and group fairness. We propose a novel uncertainty-aware evaluation methodology for RecLLMs, present empirical insights from deep uncertainty case studies, and introduce a personality profile-informed fairness benchmark that advances explainability and equity in LLM recommendations. Together, these contributions establish a foundation for safer, more interpretable RecLLMs and motivate future work on multi-model benchmarks and adaptive calibration for trustworthy deployment.

Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems

TL;DR

The paper addresses the dual challenges of predictive uncertainty and fairness in LLM-based recommendations (RecLLMs). It proposes an uncertainty-aware evaluation framework and a benchmark dataset annotated for eight demographic attributes across movies and music, enabling multi-faceted bias and reliability analyses. Using Google Gemini 1.5 Flash, it uncovers systematic fairness gaps (e.g., SNSR = , SNSV = ) that persist under prompt perturbations, and introduces personality-aware fairness metrics to reveal bias linked to user traits. The work lays a foundation for safer, more interpretable RecLLMs by integrating uncertainty quantification with fairness, and outlines directions toward multi-model benchmarks and adaptive calibration for trustworthy deployment.

Abstract

Large language models (LLMs) enable powerful zero-shot recommendations by leveraging broad contextual knowledge, yet predictive uncertainty and embedded biases threaten reliability and fairness. This paper studies how uncertainty and fairness evaluations affect the accuracy, consistency, and trustworthiness of LLM-generated recommendations. We introduce a benchmark of curated metrics and a dataset annotated for eight demographic attributes (31 categorical values) across two domains: movies and music. Through in-depth case studies, we quantify predictive uncertainty (via entropy) and demonstrate that Google DeepMind's Gemini 1.5 Flash exhibits systematic unfairness for certain sensitive attributes; measured similarity-based gaps are SNSR at 0.1363 and SNSV at 0.0507. These disparities persist under prompt perturbations such as typographical errors and multilingual inputs. We further integrate personality-aware fairness into the RecLLM evaluation pipeline to reveal personality-linked bias patterns and expose trade-offs between personalization and group fairness. We propose a novel uncertainty-aware evaluation methodology for RecLLMs, present empirical insights from deep uncertainty case studies, and introduce a personality profile-informed fairness benchmark that advances explainability and equity in LLM recommendations. Together, these contributions establish a foundation for safer, more interpretable RecLLMs and motivate future work on multi-model benchmarks and adaptive calibration for trustworthy deployment.
Paper Structure (14 sections, 3 equations, 4 figures, 4 tables)

This paper contains 14 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Uncertainty of Large Language Models.
  4. Fairness in LLM-Based Recommendation.
  5. Methodology
  6. Experimental Setup
  7. Prompt Design for Fairness and Uncertainty Assessment
  8. RESULTS AND ANALYSIS
  9. RQ1: Uncertainty Reliability Impact.
  10. RQ2: Robustness of Unfairness in LLM Recommendations.
  11. Future Directions
  12. Threats to Validity
  13. Conclusion
  14. Acknowledgments

Figures (4)

  • Figure 1: Illustrates how uncertainty in deep learning models affects recommendation reliability, using probability estimates and explanations to highlight challenges in recognizing unfamiliar inputs.
  • Figure 2: Proposed Framework for Enhancing Uncertainty Quantification and Fairness in Training LLM-based Recommendation Systems
  • Figure 3: Evaluation of LLM-Generated Music Recommendations Using Gemini 1.5 Flash: Sensitivity to Prompt Variations.
  • Figure 4: Fairness evaluation of Gemini when there appear typos in sensitive attributes (a) or when using English and French prompts (b).