Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distillation-based Scenario-Adaptive Mixture-of-Experts for the Matching Stage of Multi-scenario Recommendation

Ruibing Wang, Shuhan Guo, Haotong Du, Quanming Yao

TL;DR

This paper tackles the challenge of applying Multi-gate Mixture-of-Experts (MMOE) to the scalable matching stage in multi-scenario recommendation, where independent two-tower encoders and data imbalance hinder performance. It introduces Distillation-based Scenario-Adaptive Mixture-of-Experts (DSMOE), combining a lightweight Scenario-Adaptive Projection (SAP) module with a cross-architecture distillation framework from a joint-user-item teacher to the two-tower student. SAP injects scenario-specific, low-rank parameterizations to prevent expert collapse in tail scenarios, while knowledge distillation transfers high-order interaction signals to the efficient two-tower architecture. Experiments on two real-world datasets show DSMOE outperforms baselines, especially in data-sparse scenarios, with favorable efficiency metrics, validating its practicality for industrial retrieval systems and offering a path toward interaction-aware, scalable matching.

Abstract

Multi-scenario recommendation is pivotal for optimizing user experience across diverse contexts. While Multi-gate Mixture-of-Experts (MMOE) thrives in ranking, its transfer to the matching stage is hindered by the blind optimization inherent to independent two-tower architectures and the parameter dominance of head scenarios. To address these structural and distributional bottlenecks, we propose Distillation-based Scenario-Adaptive Mixture-of-Experts (DSMOE). Specially, we devise a Scenario-Adaptive Projection (SAP) module to generate lightweight, context-specific parameters, effectively preventing expert collapse in long-tail scenarios. Concurrently, we introduce a cross-architecture knowledge distillation framework, where an interaction-aware teacher guides the two-tower student to capture complex matching patterns. Extensive experiments on real-world datasets demonstrate DSMOE's superiority, particularly in significantly improving retrieval quality for under-represented, data-sparse scenarios.

Distillation-based Scenario-Adaptive Mixture-of-Experts for the Matching Stage of Multi-scenario Recommendation

TL;DR

This paper tackles the challenge of applying Multi-gate Mixture-of-Experts (MMOE) to the scalable matching stage in multi-scenario recommendation, where independent two-tower encoders and data imbalance hinder performance. It introduces Distillation-based Scenario-Adaptive Mixture-of-Experts (DSMOE), combining a lightweight Scenario-Adaptive Projection (SAP) module with a cross-architecture distillation framework from a joint-user-item teacher to the two-tower student. SAP injects scenario-specific, low-rank parameterizations to prevent expert collapse in tail scenarios, while knowledge distillation transfers high-order interaction signals to the efficient two-tower architecture. Experiments on two real-world datasets show DSMOE outperforms baselines, especially in data-sparse scenarios, with favorable efficiency metrics, validating its practicality for industrial retrieval systems and offering a path toward interaction-aware, scalable matching.

Abstract

Multi-scenario recommendation is pivotal for optimizing user experience across diverse contexts. While Multi-gate Mixture-of-Experts (MMOE) thrives in ranking, its transfer to the matching stage is hindered by the blind optimization inherent to independent two-tower architectures and the parameter dominance of head scenarios. To address these structural and distributional bottlenecks, we propose Distillation-based Scenario-Adaptive Mixture-of-Experts (DSMOE). Specially, we devise a Scenario-Adaptive Projection (SAP) module to generate lightweight, context-specific parameters, effectively preventing expert collapse in long-tail scenarios. Concurrently, we introduce a cross-architecture knowledge distillation framework, where an interaction-aware teacher guides the two-tower student to capture complex matching patterns. Extensive experiments on real-world datasets demonstrate DSMOE's superiority, particularly in significantly improving retrieval quality for under-represented, data-sparse scenarios.
Paper Structure (28 sections, 10 equations, 4 figures, 4 tables)

This paper contains 28 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Matching Problem of Multi-scenario Recommendation
  4. The Multi-gate Mixture-of-Experts (MMOE) Backbone
  5. Methodology
  6. Problem Formulation
  7. Framework Overview
  8. Model Architecture
  9. Embedding Layer
  10. Scenario-Adaptive Projection Modulation(SAP)
  11. SAP based MMOE
  12. Expert Networks.
  13. Scenario-Aware Mixture.
  14. Model Optimization
  15. Comparison with Existing Works
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The framework of DSMOE. (i) The Scenario-Adaptive Projection (SAP) module, which uses low-rank decomposition to generate dynamic parameters from scenario context. (ii) The SAP-based MMOE architecture. (iii) The knowledge distillation optimization framework.
  • Figure 2: Varying the number of experts for DSMOE in KuaiRand-Pure.
  • Figure 3: Varying the number of rank R for DSMOE in KuaiRand-Pure.
  • Figure 4: Case study on the KuaiRand-Pure dataset. (a) Cosine similarity of scenario-adaptive vectors $b_s$ from the SAP module. (b) Average expert activation weights from the gating network, showing clear specialization across scenarios.