Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems

Wan Jiang, Xinyi Zang, Yudong Zhao, Yusi Zou, Yunfei Lu, Junbo Tong, Yang Liu, Ming Li, Jiani Shi, Xin Yang

TL;DR

MaRCA addresses revenue optimization in large-scale recommender systems under dynamic computation constraints by formulating the multi-stage pipeline as a cooperative multi-agent reinforcement learning problem under CTDE. It introduces the Adaptive Weighting Recurrent Q-Mixer (AWRQ-Mixer) to capture inter-stage dependencies, incorporating adaptive head weighting, variance-guided credit assignment, and softplus-based monotonicity to ensure cooperative aggregation. A predictive MPC-based Revenue-Cost Balancer proactively tunes the revenue-cost trade-off using a rolling horizon, supported by AutoBucket for accurate cost estimation. Offline experiments and a large-scale online deployment demonstrate significant revenue uplifts (e.g., 16.67%) without extra compute, validating MaRCA’s practical impact for dynamic, resource-constrained recommender systems. The work offers a scalable, production-ready framework that tightly couples learning-based coordination with model-based control to maintain stability and efficiency in real-world pipelines.

Abstract

Modern recommender systems face significant computational challenges due to growing model complexity and traffic scale, making efficient computation allocation critical for maximizing business revenue. Existing approaches typically simplify multi-stage computation resource allocation, neglecting inter-stage dependencies, thus limiting global optimality. In this paper, we propose MaRCA, a multi-agent reinforcement learning framework for end-to-end computation resource allocation in large-scale recommender systems. MaRCA models the stages of a recommender system as cooperative agents, using Centralized Training with Decentralized Execution (CTDE) to optimize revenue under computation resource constraints. We introduce an AutoBucket TestBench for accurate computation cost estimation, and a Model Predictive Control (MPC)-based Revenue-Cost Balancer to proactively forecast traffic loads and adjust the revenue-cost trade-off accordingly. Since its end-to-end deployment in the advertising pipeline of a leading global e-commerce platform in November 2024, MaRCA has consistently handled hundreds of billions of ad requests per day and has delivered a 16.67% revenue uplift using existing computation resources.

MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems

TL;DR

MaRCA addresses revenue optimization in large-scale recommender systems under dynamic computation constraints by formulating the multi-stage pipeline as a cooperative multi-agent reinforcement learning problem under CTDE. It introduces the Adaptive Weighting Recurrent Q-Mixer (AWRQ-Mixer) to capture inter-stage dependencies, incorporating adaptive head weighting, variance-guided credit assignment, and softplus-based monotonicity to ensure cooperative aggregation. A predictive MPC-based Revenue-Cost Balancer proactively tunes the revenue-cost trade-off using a rolling horizon, supported by AutoBucket for accurate cost estimation. Offline experiments and a large-scale online deployment demonstrate significant revenue uplifts (e.g., 16.67%) without extra compute, validating MaRCA’s practical impact for dynamic, resource-constrained recommender systems. The work offers a scalable, production-ready framework that tightly couples learning-based coordination with model-based control to maintain stability and efficiency in real-world pipelines.

Abstract

Modern recommender systems face significant computational challenges due to growing model complexity and traffic scale, making efficient computation allocation critical for maximizing business revenue. Existing approaches typically simplify multi-stage computation resource allocation, neglecting inter-stage dependencies, thus limiting global optimality. In this paper, we propose MaRCA, a multi-agent reinforcement learning framework for end-to-end computation resource allocation in large-scale recommender systems. MaRCA models the stages of a recommender system as cooperative agents, using Centralized Training with Decentralized Execution (CTDE) to optimize revenue under computation resource constraints. We introduce an AutoBucket TestBench for accurate computation cost estimation, and a Model Predictive Control (MPC)-based Revenue-Cost Balancer to proactively forecast traffic loads and adjust the revenue-cost trade-off accordingly. Since its end-to-end deployment in the advertising pipeline of a leading global e-commerce platform in November 2024, MaRCA has consistently handled hundreds of billions of ad requests per day and has delivered a 16.67% revenue uplift using existing computation resources.
Paper Structure (34 sections, 18 equations, 4 figures, 6 tables, 3 algorithms)

This paper contains 34 sections, 18 equations, 4 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deterministic Optimizers for Computation Allocation
  4. Learning-Based Approaches for Computation Allocation
  5. Methodology
  6. Problem Formulation
  7. System Design
  8. Adaptive Weighting Recurrent Q-Mixer
  9. Adaptive Weighting Recurrent Q (AWRQ)
  10. Softplus-Based Monotonicity Constraints (SMC) for Cooperative Agents
  11. Variance-Guided Credit Assignment (VGCA)
  12. MPC-Based Revenue-Cost Balancer
  13. Experiments
  14. Offline Experiments
  15. AWRQ-Mixer Experiment Results
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Overview of the general architecture of recommender systems, highlighting the key stages and components involved in processing user requests.
  • Figure 2: MaRCA system architecture: multi-agent collaborative decision flow with Adaptive Weighting Recurrent Q-Mixer, AutoBucket TestBench, and MPC-Based Revenue-Cost Balancer.
  • Figure 3: Adaptive Weighting Recurrent Q-Mixer (AWRQ-Mixer) Architecture in MaRCA.
  • Figure 5: AutoBucket TestBench enables multi-stage computation cost estimation through traffic simulation, regression analysis, and sequence-aware modeling.