Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation

Pingjun Pan, Tingting Zhou, Peiyao Lu, Tingting Fei, Hongxiang Chen, Chuanjiang Luo

TL;DR

Hi-SAM tackles the tokenization–architecture gap in large-scale multimodal recommendation by introducing a two-stage framework. The DST stage uses Cross-Modal Geometric Alignment and Disentangled Modal-Residual Quantization to produce coherent, disentangled semantic IDs, while the HMAT stage employs Hierarchical RoPE and Memory-Anchor Attention to model inter- and intra-item structure with memory compression. Empirical results show consistent gains over state-of-the-art baselines, especially in cold-start settings, and a real-world deployment yields a 6.55% improvement in the core online metric along with a 35% latency reduction. The work demonstrates the practical viability of combining geometry-aware multimodal tokenization with hierarchy-aware sequence modeling for scalable, production-ready recommendations.

Abstract

Multi-modal recommendation has gained traction as items possess rich attributes like text and images. Semantic ID-based approaches effectively discretize this information into compact tokens. However, two challenges persist: (1) Suboptimal Tokenization: existing methods (e.g., RQ-VAE) lack disentanglement between shared cross-modal semantics and modality-specific details, causing redundancy or collapse; (2) Architecture-Data Mismatch: vanilla Transformers treat semantic IDs as flat streams, ignoring the hierarchy of user interactions, items, and tokens. Expanding items into multiple tokens amplifies length and noise, biasing attention toward local details over holistic semantics. We propose Hi-SAM, a Hierarchical Structure-Aware Multi-modal framework with two designs: (1) Disentangled Semantic Tokenizer (DST): unifies modalities via geometry-aware alignment and quantizes them via a coarse-to-fine strategy. Shared codebooks distill consensus while modality-specific ones recover nuances from residuals, enforced by mutual information minimization; (2) Hierarchical Memory-Anchor Transformer (HMAT): splits positional encoding into inter- and intra-item subspaces via Hierarchical RoPE to restore hierarchy. It inserts Anchor Tokens to condense items into compact memory, retaining details for the current item while accessing history only through compressed summaries. Experiments on real-world datasets show consistent improvements over SOTA baselines, especially in cold-start scenarios. Deployed on a large-scale social platform serving millions of users, Hi-SAM achieved a 6.55% gain in the core online metric.

Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation

TL;DR

Hi-SAM tackles the tokenization–architecture gap in large-scale multimodal recommendation by introducing a two-stage framework. The DST stage uses Cross-Modal Geometric Alignment and Disentangled Modal-Residual Quantization to produce coherent, disentangled semantic IDs, while the HMAT stage employs Hierarchical RoPE and Memory-Anchor Attention to model inter- and intra-item structure with memory compression. Empirical results show consistent gains over state-of-the-art baselines, especially in cold-start settings, and a real-world deployment yields a 6.55% improvement in the core online metric along with a 35% latency reduction. The work demonstrates the practical viability of combining geometry-aware multimodal tokenization with hierarchy-aware sequence modeling for scalable, production-ready recommendations.

Abstract

Multi-modal recommendation has gained traction as items possess rich attributes like text and images. Semantic ID-based approaches effectively discretize this information into compact tokens. However, two challenges persist: (1) Suboptimal Tokenization: existing methods (e.g., RQ-VAE) lack disentanglement between shared cross-modal semantics and modality-specific details, causing redundancy or collapse; (2) Architecture-Data Mismatch: vanilla Transformers treat semantic IDs as flat streams, ignoring the hierarchy of user interactions, items, and tokens. Expanding items into multiple tokens amplifies length and noise, biasing attention toward local details over holistic semantics. We propose Hi-SAM, a Hierarchical Structure-Aware Multi-modal framework with two designs: (1) Disentangled Semantic Tokenizer (DST): unifies modalities via geometry-aware alignment and quantizes them via a coarse-to-fine strategy. Shared codebooks distill consensus while modality-specific ones recover nuances from residuals, enforced by mutual information minimization; (2) Hierarchical Memory-Anchor Transformer (HMAT): splits positional encoding into inter- and intra-item subspaces via Hierarchical RoPE to restore hierarchy. It inserts Anchor Tokens to condense items into compact memory, retaining details for the current item while accessing history only through compressed summaries. Experiments on real-world datasets show consistent improvements over SOTA baselines, especially in cold-start scenarios. Deployed on a large-scale social platform serving millions of users, Hi-SAM achieved a 6.55% gain in the core online metric.
Paper Structure (31 sections, 16 equations, 5 figures, 5 tables)

This paper contains 31 sections, 16 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methodology
  4. Formulation and Framework
  5. Disentangled Semantic Tokenizer
  6. Cross-Modal Geometric Alignment (CGA)
  7. Disentangled Modal-Residual Quantization (DMRQ)
  8. Hierarchical Memory-Anchor Transformer
  9. Hierarchical Rotary Position Embedding (H-RoPE)
  10. Memory-Anchor Attention (MA-Attn)
  11. Training and Inference
  12. Training
  13. Inference Optimization
  14. Experiments
  15. Experimental Settings
  16. ...and 16 more sections

Figures (5)

  • Figure 1: The architecture of Hi-SAM, which consists of the Disentangled Semantic Tokenizer (DST) stage and the Hierarchical Memory-Anchor Transformer (HMAT) stage.
  • Figure 2: Ablation study on different modality combinations.
  • Figure 3: Layer-wise cosine similarity heatmaps of the baseline (a) and our DMRQ model (b)
  • Figure 4: Scalability analysis of Hi-SAM regarding (a) model depth, (b) sequence length, and (c) computational cost (GFLOPs).
  • Figure 5: 3D t-SNE visualizations of the latent space before (a) and after (b) alignment.