Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems

Sushant Mehta

TL;DR

This paper presents HELM, a human-centered evaluation framework for LLM-powered recommender systems that extends beyond accuracy by assessing five dimensions: Intent Alignment, Explanation Quality, Interaction Naturalness, Trust & Transparency, andFairness & Diversity. It defines concrete constructs and measurement protocols, validates the framework through expert assessments across 847 scenarios in Movies, Books, and Restaurants, and analyzes three state-of-the-art LLM-based systems (GPT-4, LLaMA-3.1, P5) against baselines. Key findings include GPT-4 delivering superior user-centric quality in most dimensions but exhibiting significant popularity bias and faithfulness gaps, while traditional metrics fail to capture these nuances. The work offers design recommendations (hybrid architectures, uncertainty calibration, diversity constraints) and provides an open-source HELM toolkit to promote reproducible, multi-dimensional evaluation in the recommender systems community.

Abstract

The integration of Large Language Models (LLMs) into recommendation systems has introduced unprecedented capabilities for natural language understanding, explanation generation, and conversational interactions. However, existing evaluation methodologies focus predominantly on traditional accuracy metrics, failing to capture the multifaceted human-centered qualities that determine the real-world user experience. We introduce \framework{} (\textbf{H}uman-centered \textbf{E}valuation for \textbf{L}LM-powered reco\textbf{M}menders), a comprehensive evaluation framework that systematically assesses LLM-powered recommender systems across five human-centered dimensions: \textit{Intent Alignment}, \textit{Explanation Quality}, \textit{Interaction Naturalness}, \textit{Trust \& Transparency}, and \textit{Fairness \& Diversity}. Through extensive experiments involving three state-of-the-art LLM-based recommenders (GPT-4, LLaMA-3.1, and P5) across three domains (movies, books, and restaurants), and rigorous evaluation by 12 domain experts using 847 recommendation scenarios, we demonstrate that \framework{} reveals critical quality dimensions invisible to traditional metrics. Our results show that while GPT-4 achieves superior explanation quality (4.21/5.0) and interaction naturalness (4.35/5.0), it exhibits a significant popularity bias (Gini coefficient 0.73) compared to traditional collaborative filtering (0.58). We release \framework{} as an open-source toolkit to advance human-centered evaluation practices in the recommender systems community.

HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems

TL;DR

This paper presents HELM, a human-centered evaluation framework for LLM-powered recommender systems that extends beyond accuracy by assessing five dimensions: Intent Alignment, Explanation Quality, Interaction Naturalness, Trust & Transparency, andFairness & Diversity. It defines concrete constructs and measurement protocols, validates the framework through expert assessments across 847 scenarios in Movies, Books, and Restaurants, and analyzes three state-of-the-art LLM-based systems (GPT-4, LLaMA-3.1, P5) against baselines. Key findings include GPT-4 delivering superior user-centric quality in most dimensions but exhibiting significant popularity bias and faithfulness gaps, while traditional metrics fail to capture these nuances. The work offers design recommendations (hybrid architectures, uncertainty calibration, diversity constraints) and provides an open-source HELM toolkit to promote reproducible, multi-dimensional evaluation in the recommender systems community.

Abstract

The integration of Large Language Models (LLMs) into recommendation systems has introduced unprecedented capabilities for natural language understanding, explanation generation, and conversational interactions. However, existing evaluation methodologies focus predominantly on traditional accuracy metrics, failing to capture the multifaceted human-centered qualities that determine the real-world user experience. We introduce \framework{} (\textbf{H}uman-centered \textbf{E}valuation for \textbf{L}LM-powered reco\textbf{M}menders), a comprehensive evaluation framework that systematically assesses LLM-powered recommender systems across five human-centered dimensions: \textit{Intent Alignment}, \textit{Explanation Quality}, \textit{Interaction Naturalness}, \textit{Trust \& Transparency}, and \textit{Fairness \& Diversity}. Through extensive experiments involving three state-of-the-art LLM-based recommenders (GPT-4, LLaMA-3.1, and P5) across three domains (movies, books, and restaurants), and rigorous evaluation by 12 domain experts using 847 recommendation scenarios, we demonstrate that \framework{} reveals critical quality dimensions invisible to traditional metrics. Our results show that while GPT-4 achieves superior explanation quality (4.21/5.0) and interaction naturalness (4.35/5.0), it exhibits a significant popularity bias (Gini coefficient 0.73) compared to traditional collaborative filtering (0.58). We release \framework{} as an open-source toolkit to advance human-centered evaluation practices in the recommender systems community.
Paper Structure (43 sections, 3 equations, 1 figure, 6 tables)

This paper contains 43 sections, 3 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human-Centered Recommender Systems
  4. LLM-Powered Recommender Systems
  5. Evaluation Beyond Accuracy
  6. Research Gap
  7. The HELM Framework
  8. Framework Design Principles
  9. Dimension 1: Intent Alignment
  10. Dimension 2: Explanation Quality
  11. Dimension 3: Interaction Naturalness
  12. Dimension 4: Trust and Transparency
  13. Dimension 5: Fairness and Diversity
  14. Aggregation and Reporting
  15. Experimental Setup
  16. ...and 28 more sections

Figures (1)

  • Figure 1: Intent Alignment construct scores. EIS: Explicit Intent Satisfaction, IIR: Implicit Intent Recognition, ICQ: Intent Clarification Quality, GCS: Goal Completion Support.