MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation

TL;DR

MTmixAtt tackles the pressing issues of manual feature engineering and limited cross-scenario transfer in industrial recommender systems by introducing AutoToken for data-driven feature grouping and MTmixAttBlock, a token-mixing Mixture-of-Experts backbone. The model unifies heterogeneous feature modeling and multi-scenario adaptation through a combination of shared dense experts, fine-grained experts, and sparse scenario-specific routing, enabling scalable deployment up to one billion parameters. Offline evaluations on the large-scale TRec dataset show MTmixAtt consistently outperforms Transformer-based and previous MoE baselines in CTR/CTCVR metrics, with power-law gains as model size increases. Online A/B tests in Meituan's production environment demonstrate meaningful business improvements in the Homepage and cross-scenario feeds, validating practical impact on user experience and revenue. Overall, MTmixAtt provides a scalable, cross-scenario unified framework for heterogeneous feature modeling in large-scale recommendations, with strong empirical and practical significance.

Abstract

Industrial recommender systems critically depend on high-quality ranking models. However, traditional pipelines still rely on manual feature engineering and scenario-specific architectures, which hinder cross-scenario transfer and large-scale deployment. To address these challenges, we propose \textbf{MTmixAtt}, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention, designed for large-scale recommendation tasks. MTmixAtt integrates two key components. The \textbf{AutoToken} module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups. The \textbf{MTmixAttBlock} module enables efficient token interaction via a learnable mixing matrix, shared dense experts, and scenario-aware sparse experts, capturing both global patterns and scenario-specific behaviors within a single framework. Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines including Transformer-based models, WuKong, HiFormer, MLP-Mixer, and RankMixer. At comparable parameter scales, MTmixAtt achieves superior CTR and CTCVR metrics; scaling to MTmixAtt-1B yields further monotonic gains. Large-scale online A/B tests validate the real-world impact: in the \textit{Homepage} scenario, MTmixAtt increases Payment PV by \textbf{+3.62\%} and Actual Payment GTV by \textbf{+2.54\%}. Overall, MTmixAtt provides a unified and scalable solution for modeling arbitrary heterogeneous features across scenarios, significantly improving both user experience and commercial outcomes.

Figures (8)

• Figure 1: Typical ranking models (top) rely on manual feature grouping and scenario-specific networks, leading to poor cross-scenario transferability. MTmixAtt (bottom) introduces automatic grouping and a unified architecture, enabling arbitrary feature modeling and fast cross-scenario adaptation.
• Figure 2: The architecture of MTmixAtt.
• Figure 3: Illustration of the Shared Dense MoE and Scenario-Specific Sparse MoE. (a) Overall MoE architecture. (b) Shared Dense MoE under the main scenario. (c) Scenario-Specific Sparse MoE across multiple scenarios.
• Figure 4: Effects of AutoToken and the transformation matrix in token mixing module.
• Figure 5: Ablation experiments on dense MoE in the Homepage scenario.