Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maximum In-Support Return Modeling for Dynamic Recommendation with Language Model Prior

Xiaocong Chen, Siyu Wang, Lina Yao

TL;DR

MDT4Rec addresses sub-optimal offline feedback and complex user-item interactions in sequential recommendations by integrating a large language model prior, an adaptive history-length mechanism, and efficient fine-tuning via LoRA. It shifts trajectory stitching to action inference, uses expectile regression to maximize in-support returns, and leverages MLP embeddings to enhance representation. The approach yields state-of-the-art results on five public datasets and in online simulation, with robust performance across different LLM backbones and ablations showing the value of each component. This work demonstrates a practical path for uniting LLM priors with offline reinforcement learning to deliver more accurate and efficient recommendations in dynamic settings.

Abstract

Reinforcement Learning-based recommender systems (RLRS) offer an effective way to handle sequential recommendation tasks but often face difficulties in real-world settings, where user feedback data can be sub-optimal or sparse. In this paper, we introduce MDT4Rec, an offline RLRS framework that builds on the Decision Transformer (DT) to address two major challenges: learning from sub-optimal histories and representing complex user-item interactions. First, MDT4Rec shifts the trajectory stitching procedure from the training phase to action inference, allowing the system to shorten its historical context when necessary and thereby ignore negative or unsuccessful past experiences. Second, MDT4Rec initializes DT with a pre-trained large language model (LLM) for knowledge transfer, replaces linear embedding layers with Multi-Layer Perceptrons (MLPs) for more flexible representations, and employs Low-Rank Adaptation (LoRA) to efficiently fine-tune only a small subset of parameters. We evaluate MDT4Rec on five public datasets and in an online simulation environment, demonstrating that it outperforms existing methods.

Maximum In-Support Return Modeling for Dynamic Recommendation with Language Model Prior

TL;DR

MDT4Rec addresses sub-optimal offline feedback and complex user-item interactions in sequential recommendations by integrating a large language model prior, an adaptive history-length mechanism, and efficient fine-tuning via LoRA. It shifts trajectory stitching to action inference, uses expectile regression to maximize in-support returns, and leverages MLP embeddings to enhance representation. The approach yields state-of-the-art results on five public datasets and in online simulation, with robust performance across different LLM backbones and ablations showing the value of each component. This work demonstrates a practical path for uniting LLM priors with offline reinforcement learning to deliver more accurate and efficient recommendations in dynamic settings.

Abstract

Reinforcement Learning-based recommender systems (RLRS) offer an effective way to handle sequential recommendation tasks but often face difficulties in real-world settings, where user feedback data can be sub-optimal or sparse. In this paper, we introduce MDT4Rec, an offline RLRS framework that builds on the Decision Transformer (DT) to address two major challenges: learning from sub-optimal histories and representing complex user-item interactions. First, MDT4Rec shifts the trajectory stitching procedure from the training phase to action inference, allowing the system to shorten its historical context when necessary and thereby ignore negative or unsuccessful past experiences. Second, MDT4Rec initializes DT with a pre-trained large language model (LLM) for knowledge transfer, replaces linear embedding layers with Multi-Layer Perceptrons (MLPs) for more flexible representations, and employs Low-Rank Adaptation (LoRA) to efficiently fine-tune only a small subset of parameters. We evaluate MDT4Rec on five public datasets and in an online simulation environment, demonstrating that it outperforms existing methods.
Paper Structure (21 sections, 1 theorem, 13 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 1 theorem, 13 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Formulation
  4. Decision Transformer
  5. Methodology
  6. Maximum In-Support Return Modeling
  7. Action Inference
  8. Large Language Model as Prior
  9. Training Objective
  10. Experiments
  11. Datasets and Environments
  12. Baselines and Experimental Setup
  13. Overall Comparison (RQ1)
  14. Different LLMs for Weight Initialization (RQ2)
  15. Fine-Tuning Strategies and Embedding (RQ3)
  16. ...and 6 more sections

Key Result

theorem 1

Let $\mathbf{S}_t \coloneqq \bigl[ s_{t-T+1}, R_{t-T+1}, a_{t-T+1} \,,\, s_t \bigr].$ For $\alpha \in (0,1)$, define $\mathbf{g}^\alpha(\mathbf{S}_t) \;=\; \pi^*_{\theta}(\mathbf{S}_t),$ where $\pi^*_{\theta} \;=\; \arg \min_{m} \mathcal{L}_2^\alpha(m)$, and Assume $R^{\max}_t = \max_{\tau_T\sim \mathcal{D}} R_t(\tau_T)$ is finite. Then where $R^{\max}_t$ is the maximum returns-to-go over the of

Figures (2)

  • Figure 1: The overall structure of the proposed MDT4Rec
  • Figure 2: (a). History Length study. (b). Step Size $\delta$ Study. (c). Language Auxiliary Loss Study

Theorems & Definitions (1)

  • theorem 1