Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation

Abstract

Textual explanations, generated with large language models (LLMs), are increasingly used to justify recommendations. Yet, evaluating these explanations remains a critical challenge. We advocate a shift in objective: rank, don't generate. We formalize explainable recommendation as a statement-level ranking problem, where systems rank candidate explanatory statements derived from reviews and return the top-k as explanation. This formulation mitigates hallucination by construction and enables fine-grained factual analysis. It also models factor importance through relevance scores and supports standardized, reproducible evaluation with established ranking metrics. Meaningful assessment, however, requires each statement to be explanatory (item facts affecting user experience), atomic (one opinion about one aspect), and unique (paraphrases consolidated), which is challenging to obtain from noisy reviews. We address this with (i) an LLM-based extraction pipeline producing explanatory and atomic statements, and (ii) a scalable, semantic clustering method consolidating paraphrases to enforce uniqueness. Building on this pipeline, we introduce StaR, a benchmark for statement ranking in explainable recommendation, constructed from four Amazon Reviews 2014 product categories. We evaluate popularity-based baselines and state-of-the-art models under global-level (all statements) and item-level (target item statements) ranking. Popularity baselines are competitive in global-level ranking but outperform state-of-the-art models on average in item-level ranking, exposing critical limitations in personalized explanation ranking.

Figures (4)

• Figure 1: Statement extraction and verification pipeline. (1) An LLM extracts candidate statements from a raw review. (2) A second LLM filters non-explanatory, non-atomic, or redundant statements, retaining only compliant outputs.
• Figure 2: Statement clustering pipeline. (1) ANN retrieves the top-$K$ semantically similar candidates per statement using dense embeddings. (2) A cross-encoder re-evaluates candidate pairs and retains only high-confidence matches. (3) Validated pairs define a similarity graph; connected components form initial clusters, which are refined by splitting low-cohesion components.
• Figure 3: Impact of the $\mu$ parameter on BPER+ performance across datasets.
• Figure 4: Impact of the number of layers $L$ on ExpGCN performance across datasets.