Markovian Pre-Trained Transformer for Next-Item Recommendation

TL;DR

The paper addresses the cost and heterogeneity challenges of pre-training for next-item recommendation by proposing a Markovian Pre-trained Transformer (MPT) pre-trained on synthetic Markov chains. It argues that a universal model can summarize user history while prioritizing the last interaction, and demonstrates this with a next-state NSP-style objective and a lightweight adaptor for downstream tasks. The key contributions include identifying two transferable capabilities for next-item prediction, validating universal transferability across five datasets, and showing data-efficient, fast adaptation that often surpasses language-model baselines in both accuracy and practicality. The findings establish a new paradigm: synthetic data can stimulate the essential capabilities for transferable recommendations, enabling a compact, efficient, and broadly applicable universal recommender with strong practical impact.

Abstract

We introduce the Markovian Pre-trained Transformer (MPT) for next-item recommendation, a transferable model fully pre-trained on synthetic Markov chains, yet capable of achieving state-of-the-art performance by fine-tuning a lightweight adaptor. This counterintuitive success stems from the observation of the `Markovian' nature: advanced sequential recommenders coincidentally rely on the latest interaction to make predictions, while the historical interactions serve mainly as auxiliary cues for inferring the user's general, non-sequential identity. This characteristic necessitates the capabilities of a universal recommendation model to effectively summarize the user sequence, with particular emphasis on the latest interaction. MPT inherently has the potential to be universal and transferable. On the one hand, when trained to predict the next state of Markov chains, it acquires the capabilities to estimate transition probabilities from the context (one adaptive manner for summarizing sequences) and attend to the last state to ensure accurate state transitions. On the other hand, unlike the heterogeneous interaction data, an unlimited amount of controllable Markov chains is available to boost the model capacity. We conduct extensive experiments on five public datasets from three distinct platforms to validate the superiority of Markovian pre-training over traditional recommendation pre-training and recent language pre-training paradigms.

Figures (7)

• Figure 1: Comparisons of the impact of sequence ordering (during inference). $[v_1, v_2, \ldots, v_t]$: chronologically ordered sequence; $[\text{Shuffle}(v_1, v_2, \ldots), v_t]$: partially shuffled sequence; $\text{Shuffle}(v_1, v_2, \ldots, v_t)$: completely shuffled sequence. (a) Performance across various models and datasets. (b) Performance across filtering strategies, training objectives, and embedding sizes.
• Figure 2: Overview of MPT. Left: Analogous to how linguistic corpora and mathematical data stimulate language understanding and logical reasoning, we aim to identify what types of data can effectively activate the capabilities necessary for next-item recommendation. Right: Pipeline of Markovian pre-training and recommendation fine-tuning.
• Figure 3: Data scaling: how the final recommendation performance evolves as more tokens are incorporated. Top: the validation NDCG@10 performance on the Beauty, Online Retail, and Yelp datasets. Bottom: the $\mathcal{L}_{\text{NSP}}$ loss, which reflects the accuracy of MPT in predicting the next possible state.
• Figure 4: Model scaling: the validation NDCG@10 performance under varying hidden dimension sizes.
• Figure 5: Attention maps of SASRec+, Qwen2.5, and MPT over a user sequence. The inference backbones of both Qwen2.5 and MPT are not fine-tuned using any recommendation data.

Theorems & Definitions (1)

• Remark 2.1