Scenario-Wise Rec: A Multi-Scenario Recommendation Benchmark

TL;DR

Scenario-Wise Rec addresses the fragmentation in multi-scenario recommendation research by introducing a standardized MSR benchmark that unifies data processing, model interfaces, and evaluation across six public datasets and twelve models, plus industrial validation. The framework demonstrates that expert-structured and dynamically adaptive MSR models yield strong performance, especially under sparsity, and provides empirical insights into how scenario count and data characteristics affect outcomes. By releasing open-source code and tutorials, it enables reproducibility, fair comparisons, and accelerated progress in MSR research, with practical relevance demonstrated in an industrial advertising dataset. Overall, the work establishes a practical, scalable benchmark ecosystem for MSR that supports rigorous cross-model benchmarking and real-world applicability.

Abstract

Multi Scenario Recommendation (MSR) tasks, referring to building a unified model to enhance performance across all recommendation scenarios, have recently gained much attention. However, current research in MSR faces two significant challenges that hinder the field's development: the absence of uniform procedures for multi-scenario dataset processing, thus hindering fair comparisons, and most models being closed-sourced, which complicates comparisons with current SOTA models. Consequently, we introduce our benchmark, \textbf{Scenario-Wise Rec}, which comprises 6 public datasets and 12 benchmark models, along with a training and evaluation pipeline. Additionally, we validated the benchmark using an industrial advertising dataset, reinforcing its reliability and applicability in real-world scenarios. We aim for this benchmark to offer researchers valuable insights from prior work, enabling the development of novel models based on our benchmark and thereby fostering a collaborative research ecosystem in MSR. Our source code is also publicly available.

Figures (3)

• Figure 1: An MSR example in business application: multi-scenario advertising recommendations from real world. Each slot is treated as a specific scenario in modeling.
• Figure 2: Overall pipeline of Scenario-Wise Rec.
• Figure 3: Performance versus number of scenarios on Scenario-0# and Scenario-2#.