Shielded RecRL: Explanation Generation for Recommender Systems without Ranking Degradation

TL;DR

shielded RecRL addresses the challenge of adding high quality explanations to recommendations without harming core ranking performance. By separating the ranking and explanation generation into a gradient-shielded two-tower architecture and training only the explanation tower with PPO and KL regularization, the method preserves ranking while optimizing explanations. The composite reward design—balancing explanation length, content relevance, and coherence—drives meaningful, personalized explanations, while LoRA enables parameter-efficient fine-tuning of a large language model. Experiments on a real-world Amazon Books subset show a 22.5% relative CTR improvement with no measurable degradation in ranking metrics, supported by ablations that validate the importance of the KL constraint and training dynamics. This work demonstrates that explanations can enhance user engagement without compromising recommendation accuracy, offering a practical path to interpretable, user-centric recommender systems.

Abstract

We introduce Shielded RecRL, a reinforcement learning approach to generate personalized explanations for recommender systems without sacrificing the system's original ranking performance. Unlike prior RLHF-based recommender methods that directly optimize item rankings, our two-tower architecture keeps the recommender's ranking model intact while a language model learns to produce helpful explanations. We design a composite reward signal combining explanation length, content relevance, and coherence, and apply proximal policy optimization (PPO) with a KL-divergence constraint to fine-tune a large language model with only 0.4% of its parameters trainable via LoRA adapters. In experiments on an Amazon Books dataset (approximately 50K interactions in the fantasy and romance genres), Shielded RecRL improved the relative click-through rate (CTR) by 22.5% (1.225x over baseline) while keeping the recommender's item-ranking behavior virtually unchanged. An extensive ablation study confirms that our gradient shielding strategy and reward design effectively balance explanation quality and policy drift. Our results demonstrate that Shielded RecRL enhances user-facing aspects of recommendations through rich, personalized explanations without degrading core recommendation accuracy.

Figures (5)

• Figure 1: Shielded RecRL two-tower architecture. The Recommendation Tower (left) produces a top-$K$ ranked list for the user and remains frozen ($\nabla_{\phi}=0$). The Explanation Tower (right) is a LoRA-adapted LLM that generates explanations conditioned on the top-$K$ items (dashed arrow) and the user profile. During training, only the LoRA/LLM parameters are updated ($\nabla_{\theta}$), preserving the recommender's ranking logic.
• Figure 2: Training progression showing (a) steady improvement in relative CTR reaching 22.5% increase at epoch 7, and (b) KL divergence demonstrating controlled policy drift that stabilizes due to regularization.
• Figure 3: Examples of explanations before and after Shielded RecRL fine-tuning. Before training, explanations are generic and uninformative. After training, they become personalized, referencing the user's reading history and highlighting specific appealing aspects of the recommended books.
• Figure 4: Reward component analysis showing (a) individual component evolution and (b) total reward progression. Length reward improves most significantly while coherence decreases as the model prioritizes informativeness over perfect grammar.
• Figure 5: Ablation study results showing (a) reward optimization and (b) policy drift across configurations. Removing KL regularization achieves highest reward but largest drift, while more training steps provide optimal stability-performance balance.