Towards Trustworthy LLM-Based Recommendation via Rationale Integration

TL;DR

This paper tackles the transparency-trust deficit in recommender systems by introducing an LLM-based recommender (LLM-Rec) that generates logically grounded rationales before predicting items. The approach combines rationale-enriched data construction with rationale-aware instruction tuning, enforcing a rationale-first decoding that embeds reasoning in CoT-style <think> blocks before the <item> prediction. Across two Amazon Review domains (Grocery and Office), LLM-Rec achieves significant improvements over strong baselines and demonstrates that rationales enhance both interpretability and predictive performance. A production online A/B test shows substantial CTR gains with rationales, and the authors publish a rationale-augmented dataset to support reproducibility and future work.

Abstract

Traditional recommender systems (RS) have been primarily optimized for accuracy and short-term engagement, often overlooking transparency and trustworthiness. Recently, platforms such as Amazon and Instagram have begun providing recommendation rationales to users, acknowledging their critical role in fostering trust and enhancing engagement; however, most existing systems still treat them as post-hoc artifacts. We propose an LLM-based recommender (LLM-Rec) that not only predicts items but also generates logically grounded rationales. Our approach leverages a self-annotated rationale dataset and instruction tuning in a rationale-first format, where the model generates an explanation before outputting the recommended item. By adopting this strategy and representing rationales in a chain-of-thought (CoT) style, LLM-Rec strengthens both interpretability and recommendation performance. Experiments on the Fashion and Scientific domains of the Amazon Review dataset demonstrate significant improvements over well-established baselines. To encourage reproducibility and future research, we publicly release a rationale-augmented recommendation dataset containing user histories, rationales, and recommended items.

Figures (2)

• Figure 1: GPT-4o Evaluation of Rationale Quality
• Figure 2: Comparison of three training–inference variants with/without rationale