TaoSearchEmb: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Search
Xingxian Liu, Dongshuai Li, Tao Wen, Jiahui Wan, Gui Ling, Fuyu Lv, Dan Ou, Haihong Tang
TL;DR
The paper tackles dense retrieval in large-scale e-commerce by addressing the inefficiencies of offline hard negative mining and the seesaw effect in multi-task optimization. It introduces Retrieval-GRPO, a multi-objective reinforcement learning framework that dynamically retrieves Top-K candidates, uses TaoSR1 as a real-time relevance reward, and fuses relevance, quality, and exclusivity into a GRPO-based training objective. The approach eliminates static hard negatives, enables real-time feedback from the latest model, and outperforms strong baselines in offline tests while delivering significant online gains, culminating in deployment on Taobao. The results demonstrate improved semantic generalization, especially for long-tail queries, and show that a unified multi-objective reward framework can effectively balance user satisfaction, product quality, and system-level diversity. Overall, Retrieval-GRPO accelerates iteration speed, enhances retrieval quality, and proves practical for industrial-scale e-commerce search.
Abstract
Dense retrieval, as the core component of e-commerce search engines, maps user queries and items into a unified semantic space through pre-trained embedding models to enable large-scale real-time semantic retrieval. Despite the rapid advancement of LLMs gradually replacing traditional BERT architectures for embedding, their training paradigms still adhere to BERT-like supervised fine-tuning and hard negative mining strategies. This approach relies on complex offline hard negative sample construction pipelines, which constrain model iteration efficiency and hinder the evolutionary potential of semantic representation capabilities. Besides, existing multi-task learning frameworks face the seesaw effect when simultaneously optimizing semantic relevance and non-relevance objectives. In this paper, we propose Retrieval-GRPO, a multi-objective reinforcement learning-based dense retrieval framework designed to address these challenges. The method eliminates offline hard negative sample construction by dynamically retrieving Top-K candidate products for each query during training, while introducing a relevance LLM as a reward model to generate real-time feedback. Specifically, the retrieval model dynamically optimizes embedding representations through reinforcement learning, with reward signals combining LLM-generated relevance scores, product quality scores, and multi-way exclusivity metrics to achieve multi-objective user preference alignment and real-time error correction. This mechanism not only removes dependency on hard negatives but also mitigates the seesaw effect through collaborative multi-objective optimization, significantly enhancing the model's semantic generalization capability for complex long-tail queries. Extensive offline and online experiments validate the effectiveness of Retrieval-GRPO, which has been deployed on China's largest e-commerce platform.