Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation

Ziyan Wang, Yingpeng Du, Zhu Sun, Haoyan Chua, Kaidong Feng, Wenya Wang, Jie Zhang

TL;DR

This work introduces Re2LLM, a reflective reinforcement framework for session-based recommendation that leverages LLM self-reflection to extract a pool of specialized hints and a lightweight retrieval agent trained with Proximal Policy Optimization to select useful hints for prompting. The Reflective Exploration Module builds a hint knowledge base by analyzing LLM errors across sessions, while the Reinforcement Utilization Module uses a Markov Decision Process to guide the LLM via task-specific feedback without fine-tuning the backbone. Empirical results on Hetrec2011-Movielens and Amazon Game show strong gains in few-shot settings and competitive performance against full-data baselines, with ablations confirming the value of each component and reward design. The approach reduces reliance on costly fine-tuning and demonstrates a practical path to enhance LLM-driven SBR through retrieval-augmented prompts.

Abstract

Large Language Models (LLMs) are emerging as promising approaches to enhance session-based recommendation (SBR), where both prompt-based and fine-tuning-based methods have been widely investigated to align LLMs with SBR. However, the former methods struggle with optimal prompts to elicit the correct reasoning of LLMs due to the lack of task-specific feedback, leading to unsatisfactory recommendations. Although the latter methods attempt to fine-tune LLMs with domain-specific knowledge, they face limitations such as high computational costs and reliance on open-source backbones. To address such issues, we propose a Reflective Reinforcement Large Language Model (Re2LLM) for SBR, guiding LLMs to focus on specialized knowledge essential for more accurate recommendations effectively and efficiently. In particular, we first design the Reflective Exploration Module to effectively extract knowledge that is readily understandable and digestible by LLMs. To be specific, we direct LLMs to examine recommendation errors through self-reflection and construct a knowledge base (KB) comprising hints capable of rectifying these errors. To efficiently elicit the correct reasoning of LLMs, we further devise the Reinforcement Utilization Module to train a lightweight retrieval agent. It learns to select hints from the constructed KB based on the task-specific feedback, where the hints can serve as guidance to help correct LLMs reasoning for better recommendations. Extensive experiments on multiple real-world datasets demonstrate that our method consistently outperforms state-of-the-art methods.

Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation

TL;DR

This work introduces Re2LLM, a reflective reinforcement framework for session-based recommendation that leverages LLM self-reflection to extract a pool of specialized hints and a lightweight retrieval agent trained with Proximal Policy Optimization to select useful hints for prompting. The Reflective Exploration Module builds a hint knowledge base by analyzing LLM errors across sessions, while the Reinforcement Utilization Module uses a Markov Decision Process to guide the LLM via task-specific feedback without fine-tuning the backbone. Empirical results on Hetrec2011-Movielens and Amazon Game show strong gains in few-shot settings and competitive performance against full-data baselines, with ablations confirming the value of each component and reward design. The approach reduces reliance on costly fine-tuning and demonstrates a practical path to enhance LLM-driven SBR through retrieval-augmented prompts.

Abstract

Large Language Models (LLMs) are emerging as promising approaches to enhance session-based recommendation (SBR), where both prompt-based and fine-tuning-based methods have been widely investigated to align LLMs with SBR. However, the former methods struggle with optimal prompts to elicit the correct reasoning of LLMs due to the lack of task-specific feedback, leading to unsatisfactory recommendations. Although the latter methods attempt to fine-tune LLMs with domain-specific knowledge, they face limitations such as high computational costs and reliance on open-source backbones. To address such issues, we propose a Reflective Reinforcement Large Language Model (Re2LLM) for SBR, guiding LLMs to focus on specialized knowledge essential for more accurate recommendations effectively and efficiently. In particular, we first design the Reflective Exploration Module to effectively extract knowledge that is readily understandable and digestible by LLMs. To be specific, we direct LLMs to examine recommendation errors through self-reflection and construct a knowledge base (KB) comprising hints capable of rectifying these errors. To efficiently elicit the correct reasoning of LLMs, we further devise the Reinforcement Utilization Module to train a lightweight retrieval agent. It learns to select hints from the constructed KB based on the task-specific feedback, where the hints can serve as guidance to help correct LLMs reasoning for better recommendations. Extensive experiments on multiple real-world datasets demonstrate that our method consistently outperforms state-of-the-art methods.
Paper Structure (27 sections, 6 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 6 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Session-based Recommendation
  4. Large Language Model for Recommendation
  5. Self-reflection in Large Language Models
  6. Methodology
  7. Reflective Exploration Module
  8. Multi-Round Self-Reflection Generation
  9. Automated Filtering
  10. Reinforcement Utilization Module
  11. Markov Decision Process (MDP)
  12. PPO Training
  13. Retrieval-Enhanced LLMs for Recommendation
  14. Experimental Results
  15. Experimental Setup
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Our method (c) obtains effective task-specific feedback compared with prompt-based (a) and fine-tuning-based (b) methods.
  • Figure 2: The overall architecture of our proposed Re2LLM approach. The overall flowchart (green) contains a Reflective Exploration Module (yellow) for the generation of session-aware hints as specialized knowledge and a Reinforcement Utilization Module (blue) for the learning to retrieve obtained specialized knowledge.
  • Figure 3: Performance of the proposed method with varying knowledge base size. Darker color indicates higher value.
  • Figure 4: Performance of the representative baselines and our method with varying few-shot training size for retrieval agent. The dotted lines indicate w/o-DRL(RAN) performance.