Modeling Cascaded Delay Feedback for Online Net Conversion Rate Prediction: Benchmark, Insights and Solutions

TL;DR

NetCVR prediction is challenged by multi-stage cascaded delays from click to purchase and potential refunds. The authors introduce CASCADE, the first large open dataset for online continuous NetCVR, and TESLA, an online framework with a CVR-RFR cascade, stage-wise debiasing, and delay-aware ranking to tackle delayed feedback. TESLA demonstrates robust improvements over strong baselines on CASCADE, including a notable relative RI-AUC boost for NetCVR, and provides insights into the benefits of cascaded modeling and delay signals for real-time recommender systems. The work offers a practical benchmark and methodology for accurate, online NetCVR estimation with open data to accelerate future research and deployment.

Abstract

In industrial recommender systems, conversion rate (CVR) is widely used for traffic allocation, but it fails to fully reflect recommendation effectiveness because it ignores refund behavior. To better capture true user satisfaction and business value, net conversion rate (NetCVR), defined as the probability that a clicked item is purchased and not refunded, has been proposed.Unlike CVR, NetCVR prediction involves a more complex multi-stage cascaded delayed feedback process. The two cascaded delays from click to conversion and from conversion to refund have opposite effects, making traditional CVR modeling methods inapplicable. Moreover, the lack of open-source datasets and online continuous training schemes further hinders progress in this area.To address these challenges, we introduce CASCADE (Cascaded Sequences of Conversion and Delayed Refund), the first large-scale open dataset derived from the Taobao app for online continuous NetCVR prediction. Through an in-depth analysis of CASCADE, we identify three key insights: (1) NetCVR exhibits strong temporal dynamics, necessitating online continuous modeling; (2) cascaded modeling of CVR and refund rate outperforms direct NetCVR modeling; and (3) delay time, which correlates with both CVR and refund rate, is an important feature for NetCVR prediction.Based on these insights, we propose TESLA, a continuous NetCVR modeling framework featuring a CVR-refund-rate cascaded architecture, stage-wise debiasing, and a delay-time-aware ranking loss. Extensive experiments demonstrate that TESLA consistently outperforms state-of-the-art methods on CASCADE, achieving absolute improvements of 12.41 percent in RI-AUC and 14.94 percent in RI-PRAUC on NetCVR prediction. The code and dataset are publicly available at https://github.com/alimama-tech/NetCVR.

Figures (9)

• Figure 1: Illustration of Net Conversion and Conversion. Net conversion shows multi-stage cascaded delay nature, involving Click $\to$ Conversion and Conversion $\to$ Refund delays.
• Figure 2: Conversion Rate and Net Conversion Rate by Hours.
• Figure 3: Average Refund Rate and Net Conversion Rate Across Conversion Rate Bins.
• Figure 4: Framework of TESLA. i denotes an immediate response while d denotes a delayed response.
• Figure 5: Average conversion and refund delay across CVR and refund rate groups.