Could Small Language Models Serve as Recommenders? Towards Data-centric Cold-start Recommendations

TL;DR

This work addresses the system cold-start problem in recommender systems by introducing PromptRec, which casts user–item profiles into natural language and reframes recommendations as sentiment-prediction tasks within an in-context learning paradigm. To overcome the latency of large language models, the authors develop a data-centric pipeline (RCMP) and a transferable prompt pre-training strategy (TPPT) to empower small language models, guided by a theoretical HMM-based framing that explains in-context recommendations as estimating sentiment-conditioned word likelihoods. The paper introduces a cold-start benchmark and demonstrates that refined pre-training on a targeted corpus and cross-domain prompt transfer enable small LMs (e.g., BERT-mini) to achieve performance comparable to large models with far lower inference costs, including substantial speedups. These findings offer practical pathways for deploying personalization in startup or latency-constrained environments and establish a benchmark to spur further research on system cold-start recommendations.

Abstract

Recommendation systems help users find matched items based on their previous behaviors. Personalized recommendation becomes challenging in the absence of historical user-item interactions, a practical problem for startups known as the system cold-start recommendation. While existing research addresses cold-start issues for either users or items, we still lack solutions for system cold-start scenarios. To tackle the problem, we propose PromptRec, a simple but effective approach based on in-context learning of language models, where we transform the recommendation task into the sentiment analysis task on natural language containing user and item profiles. However, this naive approach heavily relies on the strong in-context learning ability emerged from large language models, which could suffer from significant latency for online recommendations. To solve the challenge, we propose to enhance small language models for recommender systems with a data-centric pipeline, which consists of: (1) constructing a refined corpus for model pre-training; (2) constructing a decomposed prompt template via prompt pre-training. They correspond to the development of training data and inference data, respectively. The pipeline is supported by a theoretical framework that formalizes the connection between in-context recommendation and language modeling. To evaluate our approach, we introduce a cold-start recommendation benchmark, and the results demonstrate that the enhanced small language models can achieve comparable cold-start recommendation performance to that of large models with only $17\%$ of the inference time. To the best of our knowledge, this is the first study to tackle the system cold-start recommendation problem. We believe our findings will provide valuable insights for future works. The benchmark and implementations are available at https://github.com/JacksonWuxs/PromptRec.

Figures (4)

• Figure 1: Illustration of different cold-start recommendation scenarios, including user cold-start rashid2002getting, item cold-start ouyang2021learning, user-item cold-start lee2019melu, few-shot recommend cui2022m6zhang2021language, and system cold-start (ours).
• Figure 2: Refining corpus for model pre-training (RCMP) in system cold-start recommendation.
• Figure 3: Transferable prompt pre-training (TPPT) for PromptRec in system cold-start recommendation.
• Figure 4: Averaged cold-start recommendation performance compared with inference time over different model scales.