Beyond the Flat Sequence: Hierarchical and Preference-Aware Generative Recommendations

TL;DR

A novel framework built upon a two-stage paradigm that injects these crucial structural priors into the model to handle the drawback is proposed, which achieves state-of-the-art performance over multiple strong baselines, including HSTU and MTGR.

Abstract

Generative Recommenders (GRs), exemplified by the Hierarchical Sequential Transduction Unit (HSTU), have emerged as a powerful paradigm for modeling long user interaction sequences. However, we observe that their "flat-sequence" assumption overlooks the rich, intrinsic structure of user behavior. This leads to two key limitations: a failure to capture the temporal hierarchy of session-based engagement, and computational inefficiency, as dense attention introduces significant noise that obscures true preference signals within semantically sparse histories, which deteriorates the quality of the learned representations. To this end, we propose a novel framework named HPGR (Hierarchical and Preference-aware Generative Recommender), built upon a two-stage paradigm that injects these crucial structural priors into the model to handle the drawback. Specifically, HPGR comprises two synergistic stages. First, a structure-aware pre-training stage employs a session-based Masked Item Modeling (MIM) objective to learn a hierarchically-informed and semantically rich item representation space. Second, a preference-aware fine-tuning stage leverages these powerful representations to implement a Preference-Guided Sparse Attention mechanism, which dynamically constrains computation to only the most relevant historical items, enhancing both efficiency and signal-to-noise ratio. Empirical experiments on a large-scale proprietary industrial dataset from APPGallery and an online A/B test verify that HPGR achieves state-of-the-art performance over multiple strong baselines, including HSTU and MTGR.

Figures (8)

• Figure 1: A comparison of attention mechanisms visualizing how different models interpret user history. (a) The baseline HSTU model, which treats history as a "flat-sequence", exhibits scattered, item-level attention that fails to capture the underlying structure of user intent. (b) In contrast, HPGR learns to understand the hierarchical, session-based nature of user behavior.
• Figure 2: The overall architecture of our proposed HPGR framework. The model operates in a two-stage training paradigm. The central flow illustrates the shared foundation, including the Session Enhancement Module (SEM), which provides hierarchical context. The architecture then branches into two distinct stages: (1) Stage 1 (Pre-training), which uses a Transformer Encoder and a Masked Item Modeling objective to learn structurally-aware item representations. (2) Stage 2 (Fine-tuning), which employs an HSTU Encoder for the downstream pCTR task, enhanced by our novel Preference-Guided Sparse Attention (PGSA) mechanism. The panels on the right provide detailed views of the hierarchical structure of the SEM and the operational steps of PGSA. The top-left panel highlights the weight inheritance strategy, where pre-trained modules are updated with a smaller learning rate during fine-tuning.
• Figure 3: The workflow of Time-Aware Positional Encoding. First, the candidate item $c_1$ is inserted into the user's history sequence at its correct chronological position, forming a complete temporally sorted sequence. Then, a new set of positional IDs is assigned to this final sequence, ensuring that each item's position reflects its precise temporal order before being passed to the embedding layer.
• Figure 4: Performance consistency of the top models across the sampled and full test datasets.
• Figure 5: Comparison of embedding properties between Random Initialization and Pre-training. Top left: L2 norm distribution shifts to a high-density peak. Bottom row: UMAP visualization reveals that the embeddings organize into meaningful semantic patterns, in contrast to the initial random cloud.