Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation

TL;DR

The paper addresses the gap in test-time scaling for DLRS by introducing prediction merging as a way to produce diverse, meaningful outputs for the same input. It proposes two complementary strategies—heterogeneous architectures and initialization randomness—to generate diverse predictions, which are fused in the logit space. Through extensive experiments on three benchmarks with eight backbones, the approach demonstrates consistent gains, stability, and efficiency, and shows orthogonality to parameter scaling. The work also contrasts prediction merging with width/depth scaling and model merging, highlighting practical benefits for scalable online deployment. Overall, prediction merging offers a scalable, robust, and deployment-friendly avenue for improving DLRS performance without retraining larger parameter sets.

Abstract

Inspired by the success of language models (LM), scaling up deep learning recommendation systems (DLRS) has become a recent trend in the community. All previous methods tend to scale up the model parameters during training time. However, how to efficiently utilize and scale up computational resources during test time remains underexplored, which can prove to be a scaling-efficient approach and bring orthogonal improvements in LM domains. The key point in applying test-time scaling to DLRS lies in effectively generating diverse yet meaningful outputs for the same instance. We propose two ways: One is to explore the heterogeneity of different model architectures. The other is to utilize the randomness of model initialization under a homogeneous architecture. The evaluation is conducted across eight models, including both classic and SOTA models, on three benchmarks. Sufficient evidence proves the effectiveness of both solutions. We further prove that under the same inference budget, test-time scaling can outperform parameter scaling. Our test-time scaling can also be seamlessly accelerated with the increase in parallel servers when deployed online, without affecting the inference time on the user side. Code is available.

Figures (6)

• Figure 1: Comparison between Parameter Scaling and Test-time Scaling via Prediction Merging. The latter can benefit from additional improvement, superior stability, higher efficiency, and easy deployment.
• Figure 2: Visualization of different structures. (a) Classic Model design with Embedding, Interaction, and Prediction Module. (b) Depth Scabling Model design, which scales the depth of the interaction module and implicitly fuses information. (c) The Width Scaling Model design increases the number of interaction modules and the associated embedding table in parallel. It applies regularization among the interaction modules to encourage diversity and prevent overlapping. (d) Prediction Merging fuses the information directly in the logit space. For all structures, the interaction module and prediction module can have the same or different architectures. Green box indicates the information fusion process.
• Figure 3: Performance scaling curves across three datasets (Avazu, Criteo, and KDD12) as a function of multiplicative FLOPs scaling factor. Each subfigure shows a model's performance under different computational budgets. Scalable architectures (Wukong and RankMixer) exhibit consistent performance improvements following scaling laws, while others (DCNv2 and FinalMLP) demonstrate diminishing or negative returns at higher FLOPs, indicating limited parameter scalability.
• Figure 4: Efficiency Study.
• Figure 5: JS-divergence of FinalMLP models on Avazu.