LLM-Enhanced Reinforcement Learning for Long-Term User Satisfaction in Interactive Recommendation

TL;DR

Interactive recommender systems exhibit content homogeneity and filter bubbles due to short-term overfitting. LERL delivers a hierarchical framework that couples an LLM-based high-level semantic planner with an RL-based low-level policy to optimize long-term user satisfaction, formalized as $\mathbb{E}_{\pi}[\sum_{t=1}^{N} \gamma^{t-1} r_t]$ and implemented by narrowing the action space to content categories. A high-level critic provides semantic reflections via generated text to guide planning, while a low-level Transformer-based actor learns fine-grained item rankings under category constraints using PPO. Across real-world datasets, LERL achieves superior long-term metrics ($T_{\text{int}}$, $R_{\text{cum}}$) compared with state-of-the-art baselines, demonstrating improved diversity and reduced overexposure in sequential recommendations and validating the practical value of integrating LLM planning with RL for interactive recommendation.

Abstract

Interactive recommender systems can dynamically adapt to user feedback, but often suffer from content homogeneity and filter bubble effects due to overfitting short-term user preferences. While recent efforts aim to improve content diversity, they predominantly operate in static or one-shot settings, neglecting the long-term evolution of user interests. Reinforcement learning provides a principled framework for optimizing long-term user satisfaction by modeling sequential decision-making processes. However, its application in recommendation is hindered by sparse, long-tailed user-item interactions and limited semantic planning capabilities. In this work, we propose LLM-Enhanced Reinforcement Learning (LERL), a novel hierarchical recommendation framework that integrates the semantic planning power of LLM with the fine-grained adaptability of RL. LERL consists of a high-level LLM-based planner that selects semantically diverse content categories, and a low-level RL policy that recommends personalized items within the selected semantic space. This hierarchical design narrows the action space, enhances planning efficiency, and mitigates overexposure to redundant content. Extensive experiments on real-world datasets demonstrate that LERL significantly improves long-term user satisfaction when compared with state-of-the-art baselines. The implementation of LERL is available at https://anonymous.4open.science/r/code3-18D3/.

Figures (5)

• Figure 1: Overview of the LERL framework.
• Figure 2: Prompt design for the high-level actor, which selects category set by integrating category-level interaction history and sampled reflections.
• Figure 3: Prompt design for the high-level critic, which generates the reflection from category-level interactions and auxiliary statistics in a user's session.
• Figure 4: Recommendation trajectories of LERL and PPO.
• Figure 5: The results of the ablation study.