Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequential Regression for Continuous Value Prediction using Residual Quantization

Runpeng Cui, Zhipeng Sun, Chi Lu, Peng Jiang

TL;DR

A residual quantization (RQ)-based sequence learning framework that represents target continuous values as a sum of ordered quantization codes, predicted recursively from coarse to fine granularity with diminishing quantization errors, allowing the model to capture continuous representations for quantization codes and further improving prediction accuracy.

Abstract

Continuous value prediction plays a crucial role in industrial-scale recommendation systems, including tasks such as predicting users' watch-time and estimating the gross merchandise value (GMV) in e-commerce transactions. However, it remains challenging due to the highly complex and long-tailed nature of the data distributions. Existing generative approaches rely on rigid parametric distribution assumptions, which fundamentally limits their performance when such assumptions misalign with real-world data. Overly simplified forms cannot adequately model real-world complexities, while more intricate assumptions often suffer from poor scalability and generalization. To address these challenges, we propose a residual quantization (RQ)-based sequence learning framework that represents target continuous values as a sum of ordered quantization codes, predicted recursively from coarse to fine granularity with diminishing quantization errors. We introduce a representation learning objective that aligns RQ code embedding space with the ordinal structure of target values, allowing the model to capture continuous representations for quantization codes and further improving prediction accuracy. We perform extensive evaluations on public benchmarks for lifetime value (LTV) and watch-time prediction, alongside a large-scale online experiment for GMV prediction on an industrial short-video recommendation platform. The results consistently show that our approach outperforms state-of-the-art methods, while demonstrating strong generalization across diverse continuous value prediction tasks in recommendation systems.

Sequential Regression for Continuous Value Prediction using Residual Quantization

TL;DR

A residual quantization (RQ)-based sequence learning framework that represents target continuous values as a sum of ordered quantization codes, predicted recursively from coarse to fine granularity with diminishing quantization errors, allowing the model to capture continuous representations for quantization codes and further improving prediction accuracy.

Abstract

Continuous value prediction plays a crucial role in industrial-scale recommendation systems, including tasks such as predicting users' watch-time and estimating the gross merchandise value (GMV) in e-commerce transactions. However, it remains challenging due to the highly complex and long-tailed nature of the data distributions. Existing generative approaches rely on rigid parametric distribution assumptions, which fundamentally limits their performance when such assumptions misalign with real-world data. Overly simplified forms cannot adequately model real-world complexities, while more intricate assumptions often suffer from poor scalability and generalization. To address these challenges, we propose a residual quantization (RQ)-based sequence learning framework that represents target continuous values as a sum of ordered quantization codes, predicted recursively from coarse to fine granularity with diminishing quantization errors. We introduce a representation learning objective that aligns RQ code embedding space with the ordinal structure of target values, allowing the model to capture continuous representations for quantization codes and further improving prediction accuracy. We perform extensive evaluations on public benchmarks for lifetime value (LTV) and watch-time prediction, alongside a large-scale online experiment for GMV prediction on an industrial short-video recommendation platform. The results consistently show that our approach outperforms state-of-the-art methods, while demonstrating strong generalization across diverse continuous value prediction tasks in recommendation systems.
Paper Structure (30 sections, 18 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 30 sections, 18 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Regression by Classification
  4. Regression by Generative Approaches
  5. Method
  6. Notations
  7. Residual Quantization
  8. RQ Modeling for Continuous Value Prediction
  9. RQ codes modeling.
  10. Architecture.
  11. Objective Function
  12. RQ code generation.
  13. Regression.
  14. Continuous representation learning.
  15. Optimization and Inference
  16. ...and 15 more sections

Figures (4)

  • Figure 1: The GMV distribution in a large-scale online short-video commercial scenario. Each subfigure shows the GMV frequency distribution for value ranges of 10–20, 10–30, 30–40, and 40–50, from top to bottom.
  • Figure 2: The architecture of the proposed RQ-Reg method.
  • Figure 3: Learned representations of target values from different models on KuaiRec dataset. Colors of spots represent the magnitude of the corresponding target values from low (cool) to high (warm).
  • Figure 4: Daily ADVV gains during the online evaluation, where the A/B test period spans days 1-7, and A/A test spans days 9-13.