Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

Jianhui Yang, Yiming Jin, Pengkun Jiao, Chenhe Dong, Zerui Huang, Shaowei Yao, Xiaojiang Zhou, Dan Ou, Haihong Tang

TL;DR

TaoSR-AGRL tackles reward sparsity and exploration stagnation in large-language-model–driven e-commerce relevance by introducing Rule-aware Reward Shaping and Adaptive Guided Replay. The framework densifies supervision along the reasoning trajectory and provides on-demand, targeted ground-truth guidance to hard samples, enabling robust multi-step reasoning under complex business rules. Extensive offline evaluations show state-of-the-art improvements in relevance and rule adherence, while online side-by-side tests and deployment on Taobao demonstrate real-world impact despite initial recall-related trade-offs that were mitigated with architectural adjustments. The work provides a practical, scalable blueprint for integrating adaptive RL with LLMs in industrial search settings, balancing semantic relevance with business objectives.

Abstract

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

TL;DR

TaoSR-AGRL tackles reward sparsity and exploration stagnation in large-language-model–driven e-commerce relevance by introducing Rule-aware Reward Shaping and Adaptive Guided Replay. The framework densifies supervision along the reasoning trajectory and provides on-demand, targeted ground-truth guidance to hard samples, enabling robust multi-step reasoning under complex business rules. Extensive offline evaluations show state-of-the-art improvements in relevance and rule adherence, while online side-by-side tests and deployment on Taobao demonstrate real-world impact despite initial recall-related trade-offs that were mitigated with architectural adjustments. The work provides a practical, scalable blueprint for integrating adaptive RL with LLMs in industrial search settings, balancing semantic relevance with business objectives.

Abstract

Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.

Paper Structure

This paper contains 29 sections, 4 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Search Relevance
  4. RLVR for LLM Reasoning
  5. Data Construction
  6. Initial Filtering and Human Annotation
  7. Difficulty-Aware Dynamic Sampling
  8. Undersampling for Distribution Balancing
  9. Rule-aware Reward Shaping
  10. Adaptive Guided Replay
  11. Guide When the Model Needs
  12. Guide What the Model Needs
  13. Experiment
  14. Experimental Setup
  15. Dataset and Evaluation Metrics
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Overview of the proposed TaoSR-AGRL for e-commerce search relevance. (1) Data Construction builds a balanced and challenging dataset. (2) Rule-aware Reward Shaping decomposes sparse final rewards into dense, structured signals covering key reasoning steps to ensure faithful and interpretable reasoning. (3) Adaptive Guided Replay selectively replays low-reward, hard samples with targeted guidance to explore higher-value reasoning trajectories.
  • Figure 2: Ablation on training dynamics: Compared to its variants, TaoSR-AGRL exhibits (a) higher sample efficiency, (b) greater reward delta, and (c) stable policy entropy preservation, effectively preventing collapse.