MiniRec: Data-Efficient Reinforcement Learning for LLM-based Recommendation

TL;DR

MiniRec tackles data inefficiency in RL‑based LLM recommendation by aligning data valuation with RL dynamics. It jointly optimizes sample learnability via reward signals, representativeness via gradient alignment with an ideal optimization trajectory, and diversity, then uses curriculum scheduling to train from easy to hard samples. The method defines $L(x_i)$ with a Gaussian over proxy rewards, $R(x_i)$ via Hessian‑vector products, and a combined value $V(x_i|\\mathcal{S})$ that guides greedy subset selection, followed by curriculum partitioning. Empirical results on real‑world datasets show MiniRec achieves comparable or superior recommendation quality with far less data and training time, and transfers across different LLM architectures, underscoring its practical impact for scalable, reasoning‑aware LLM recommender systems.

Abstract

The integration of reinforcement learning (RL) into large language models (LLMs) has opened new opportunities for recommender systems by eliciting reasoning and improving user preference modeling. However, RL-based LLM recommendation faces significant efficiency challenges, making full-data training costly. Existing data selection methods define sample value based on learnability or representativeness, yet their loss- or gradient-driven or dataset coverage-driven criteria often misalign with RL learning dynamics, resulting in suboptimal performance. To address this, we propose MiniRec, a data selection framework tailored for RL-based LLM recommendation. MiniRec evaluates sample learnability using key RL signals -- rewards -- pruning samples that are too easy (too high reward) or too difficult (consistently low reward). It assesses representativeness by aligning sample gradients with the approximated "ideal" global RL optimization trajectory, selecting samples that mainly drive model updates, and it also enforces diversity to reduce redundancy. Combined with a curriculum learning strategy from easy to hard samples, MiniRec significantly reduces training cost while largely preserving performance. Extensive experiments demonstrate MiniRec's effectiveness, highlighting the importance of reward-aligned, trajectory-informed data selection in RL-based LLM recommendation.

Figures (7)

• Figure 1: Comparison of average rewards distributions before and after training. Each horizontal bar shows the proportion of samples with very low average rewards (red) and normal rewards (blue). The shaded band highlights the subset of 557 samples (27.2%) that stayed at very low rewards both before and after training, indicating that more than one quarter of the dataset consistently failed to achieve any optimization contribution throughout training.
• Figure 2: The MiniRec pipeline. We estimate sample-level learnability, representativeness, and diversity from the full training data, normalize these metrics, and integrate them through a value function for greedy subset selection. The obtained subset $\mathcal{S}$ is then partitioned for curriculum scheduling to train the model from easy to hard samples.
• Figure 3: MiniRec’s curriculum scheduling mechanism.
• Figure 4: Evaluation of MiniRec’s cross‑model generalization on the CDs & Vinyl dataset. Subsets selected using one LLM (Gemma‑2‑2B‑it) are transferred to another LLM (Qwen2.5‑3B‑Instruct) for fine‑tuning. MiniRec maintains consistently strong performance across all metrics, demonstrating its ability to capture model‑agnostic and broadly transferable data value.
• Figure 5: Training time comparison of MiniRec and full-data fine-tuning on the CDs & Vinyl and Instruments datasets under different subset sizes ($|\mathcal{S}| \in \{256, 512, 1024\}$). MiniRec achieves substantial time reduction—up to 82%—while maintaining comparable performance.