Pre-train and Fine-tune: Recommenders as Large Models

TL;DR

The paper addresses rapid, domain-specific shifts in user interests that challenge industrial recommenders. It reframes recommender systems as large pre-trained models and introduces Information-Aware Adaptive Kernel (IAK) guided by information bottleneck theory to fine-tune for downstream domains, enabling knowledge compression and task-specific knowledge matching. The approach is validated through extensive offline benchmarks on real-world datasets and multiple large-scale online A/B tests, showing consistent improvements over strong baselines and successful deployment on a billion-user platform. The work provides theoretical IB foundations for fine-tuning, practical deployment strategies, and insights into potential issues, offering a scalable, interpretable path for cross-domain recommendation at industrial scale.

Abstract

In reality, users have different interests in different periods, regions, scenes, etc. Such changes in interest are so drastic that they are difficult to be captured by recommenders. Existing multi-domain learning can alleviate this problem. However, the structure of the industrial recommendation system is complex, the amount of data is huge, and the training cost is extremely high, so it is difficult to modify the structure of the industrial recommender and re-train it. To fill this gap, we consider recommenders as large pre-trained models and fine-tune them. We first propose the theory of the information bottleneck for fine-tuning and present an explanation for the fine-tuning technique in recommenders. To tailor for recommendation, we design an information-aware adaptive kernel (IAK) technique to fine-tune the pre-trained recommender. Specifically, we define fine-tuning as two phases: knowledge compression and knowledge matching and let the training stage of IAK explicitly approximate these two phases. Our proposed approach designed from the essence of fine-tuning is well interpretable. Extensive online and offline experiments show the superiority of our proposed method. Besides, we also share unique and important lessons we learned when deploying the method in a large-scale online platform. We also present the potential issues of fine-tuning techniques in recommendation systems and the corresponding solutions. The recommender with IAK technique has been deployed on the homepage of a billion-scale online food platform for several months and has yielded considerable profits in our business.

Figures (4)

• Figure 1: Heatmap of results of control test.
• Figure 2: Results of parameter sensitivity test.
• Figure 3: Results of exploration study.
• Figure 4: Some hot items in Region4, the vertical axis represents the number of impressions of the items, and the items surrounded by green squares are the specialties of Region4.