Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-Stage Constrained Actor-Critic for Short Video Recommendation

Qingpeng Cai, Zhenghai Xue, Chi Zhang, Wanqi Xue, Shuchang Liu, Ruohan Zhan, Xueliang Wang, Tianyou Zuo, Wentao Xie, Dong Zheng, Peng Jiang, Kun Gai

TL;DR

This work tackles short-video recommendations by modeling user interactions as a Constrained Markov Decision Process, where the main objective is cumulative WatchTime under constraints from auxiliary signals like Likes and Shares. It introduces a Two-Stage Constrained Actor-Critic (TSCAC) method that first trains separate policies for each auxiliary signal and then learns a main policy that is softly regularized toward those auxiliary policies via a constrained optimization with a closed-form solution. The approach delivers superior performance offline and online, outperforming constrained RL and Pareto-based baselines, and has been deployed in a production short-video platform. The results demonstrate that multi-critic evaluation and staged learning effectively balance long-term engagement with multi-faceted user responses, providing practical benefits for real-world recommender systems.

Abstract

The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including watch time and various types of interactions with multiple videos. One the one hand, the platforms aims at optimizing the users' cumulative watch time (main goal) in long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also needs to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such like, follow, share etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms can not work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. At stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both watch time and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform.

Two-Stage Constrained Actor-Critic for Short Video Recommendation

TL;DR

This work tackles short-video recommendations by modeling user interactions as a Constrained Markov Decision Process, where the main objective is cumulative WatchTime under constraints from auxiliary signals like Likes and Shares. It introduces a Two-Stage Constrained Actor-Critic (TSCAC) method that first trains separate policies for each auxiliary signal and then learns a main policy that is softly regularized toward those auxiliary policies via a constrained optimization with a closed-form solution. The approach delivers superior performance offline and online, outperforming constrained RL and Pareto-based baselines, and has been deployed in a production short-video platform. The results demonstrate that multi-critic evaluation and staged learning effectively balance long-term engagement with multi-faceted user responses, providing practical benefits for real-world recommender systems.

Abstract

The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including watch time and various types of interactions with multiple videos. One the one hand, the platforms aims at optimizing the users' cumulative watch time (main goal) in long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also needs to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such like, follow, share etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms can not work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. At stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned at the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both watch time and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform.
Paper Structure (34 sections, 2 theorems, 14 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 34 sections, 2 theorems, 14 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Recommendation
  4. Constrained Reinforcement Learning
  5. Multi-objective Optimization
  6. Constrained Markov Decision Process for Short Video Recommendation
  7. Two-Stage Constrained Actor-Critic
  8. Multi-Critic Policy Estimation
  9. Stage One: Policy Learning for Auxiliary Responses
  10. Stage Two: Softly Constrained Optimization of the Main Response
  11. Offline Experiments
  12. Setup
  13. Dataset
  14. MDP
  15. Evaluation
  16. ...and 19 more sections

Key Result

Theorem 1

The Lagrangian of Eq. (eq:awac) has the closed form solution where $\lambda_i$ with $i=2,\dots,m$ are Lagrangian multipliers.

Figures (5)

  • Figure 1: An example of a popular short video (TikTok, Kuaishou, etc) platform.
  • Figure 2: The MDP of short video recommendation.
  • Figure 3: Effect of the value of the Lagrangian multiplier on the performance.
  • Figure 4: The workflow of RL in production system.
  • Figure 5: Online performance gap of TSCAC over the LTR baseline of each day.

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2