Beyond Static LLM Policies: Imitation-Enhanced Reinforcement Learning for Recommendation

TL;DR

This work tackles the latency and reliability challenges of deploying LLMs as direct recommendation policies by proposing IL-Rec, an offline, model-based RL framework that learns from LLM-generated demonstrations. It uses inverse reinforcement learning to extract reward models from these demonstrations and trains a reinforcement learning policy guided by the demonstrations' cumulative rewards, avoiding fine-tuning and online LLM calls. The method integrates four LLM modules (Reflector, Planner, LLM Actor, LLM Critic) to generate rich expert trajectories, and employs a discriminator-based weighting scheme to emphasize high-quality demonstrations while mitigating suboptimal artifacts. Empirical results on two benchmark datasets show that IL-Rec achieves superior long-term engagement (higher $R_{traj}$) and longer interaction horizons compared to state-of-the-art offline RL and static LLM baselines, with performance improving as LLM capabilities scale. The approach offers a practical, scalable path to semantically informed, low-latency recommendations in real-world systems.

Abstract

Recommender systems (RecSys) have become critical tools for enhancing user engagement by delivering personalized content across diverse digital platforms. Recent advancements in large language models (LLMs) demonstrate significant potential for improving RecSys, primarily due to their exceptional generalization capabilities and sophisticated contextual understanding, which facilitate the generation of flexible and interpretable recommendations. However, the direct deployment of LLMs as primary recommendation policies presents notable challenges, including persistent latency issues stemming from frequent API calls and inherent model limitations such as hallucinations and biases. To address these issues, this paper proposes a novel offline reinforcement learning (RL) framework that leverages imitation learning from LLM-generated trajectories. Specifically, inverse reinforcement learning is employed to extract robust reward models from LLM demonstrations. This approach negates the need for LLM fine-tuning, thereby substantially reducing computational overhead. Simultaneously, the RL policy is guided by the cumulative rewards derived from these demonstrations, effectively transferring the semantic insights captured by the LLM. Comprehensive experiments conducted on two benchmark datasets validate the effectiveness of the proposed method, demonstrating superior performance when compared against state-of-the-art RL-based and in-context learning baselines. The code can be found at https://github.com/ArronDZhang/IL-Rec.

Figures (7)

• Figure 1: Comparison of three recommendation strategies. RL4RS models often struggle to interpret user intent from textual input, resulting in less accurate suggestions. Static LLM policies tend to favor prevalent items of the same type, which may reduce recommendation diversity. The integration of RL into LLM recommendation exploits language understanding with adaptive and user-specific decision-making.
• Figure 2: The left part illustrates the collection of demonstrations from the LLM policy. Expert demonstrations are synthesized through the collaboration of the Planner, Reflector, LLM-Actor, and LLM-Critic. Subsequently, as shown on the right, adversarial inverse reinforcement learning (IRL) is employed to extract an effective reward function by training a discriminator to distinguish expert-generated trajectories from those produced by the current policy interacting with the learned world model. Through iterative interactions with the world model, the policy is progressively refined to surpass the original LLM expert demonstrations, while mitigating overfitting to artifacts of the simulated environment.
• Figure 3: Hyperparameter sensitivity of $\beta$, $\alpha$, $\alpha_{\mathrm{ent}}$, and $\lambda_{\mathrm{imit}}$ on two datasets.
• Figure 4: Prompt to generate the reflection from Reflector as in BiLLP shi2024billp
• Figure 5: Prompt to generate the guidance from Planner as in BiLLP shi2024billp