Reinforced Latent Reasoning for LLM-based Recommendation

TL;DR

This work addresses the cost and data requirements of explicit chain-of-thought reasoning in LLM-based recommendations by introducing LatentR$^3$, which learns compact latent reasoning tokens rather than textual CoT. The architecture adds a LatentRATT layer to generate latent tokens and combines it with a two-stage RL training regime (warm-up SFT followed by LR-GRPO using continuous, perplexity-based rewards) to optimize reasoning without CoT supervision. Empirical results on Amazon domain datasets show LatentR$^3$ yields consistent, significant gains over strong baselines, with notably better performance on long-tail items and minimal inference latency due to using only a few latent tokens. The approach offers a practical, scalable alternative to explicit CoT approaches for deploying LLM-based recommendations in real-world systems.

Abstract

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks, sparking growing interest in their application to preference reasoning in recommendation systems. Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data. However, these methods face significant practical limitations due to (1) the difficulty of obtaining high-quality CoT data in recommendation and (2) the high inference latency caused by generating CoT reasoning. In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning. This approach eliminates the need for explicit CoT generation and improves inference efficiency, as few latent tokens can effectively capture the entire reasoning process. Building on this idea, we propose \textit{\underline{R}einforced \underline{Latent} \underline{R}easoning for \underline{R}ecommendation} (LatentR$^3$), a novel end-to-end training framework that leverages reinforcement learning (RL) to optimize latent reasoning without relying on any CoT data. LatentR$^3$ adopts a two-stage training strategy: first, supervised fine-tuning to initialize the latent reasoning module, followed by pure RL training to encourage exploration through a rule-based reward design. Our RL implementation is based on a modified GRPO algorithm, which reduces computational overhead during training and introduces continuous reward signals for more efficient learning. Extensive experiments demonstrate that LatentR$^3$ enables effective latent reasoning without any direct supervision of the reasoning process, significantly improving performance when integrated with different LLM-based recommendation methods. Our codes are available at https://github.com/xuwenxinedu/R3.

Figures (8)

• Figure 1: An illustration of the model architecture and training strategy of the proposed LatentR$^3$. On the architectural side, a LatentRATT layer is introduced to generate latent reasoning continuous tokens. The training follows a two-stage framework: the first stage performs warm-up tuning via supervised fine-tuning (SFT), while the second stage applies RL based on a modified GRPO, termed LR-GRPO, to further optimize the reasoning.
• Figure 2: Performance improvement of LatentR$^3$ over BIGRec on both popular and unpopular items.
• Figure 3: Impact of Reasoning Length on LatentR$^3$ Performance.
• Figure 4: Comparison to the original GRPO regarding Performance (Left) and efficiency (Right).
• Figure 5: Inference time comparison across non-reasoning (BIGRec), LatentR$^3$ (Ours), and explicit CoT methods (COT).