MERGE: Next-Generation Item Indexing Paradigm for Large-Scale Streaming Recommendation

TL;DR

MERGE addresses the limitations of Vector Quantization in large-scale streaming recommender item indexing, where skewed and non-stationary item distributions degrade assignment accuracy, occupancy uniformity, and cluster separation. It introduces adaptive dynamic clustering from scratch, real-time occupancy monitoring, and fine-to-coarse merging to build hierarchical index structures. Offline and online experiments demonstrate that MERGE improves item-to-cluster alignment, cluster uniformity, and inter-cluster separation, with online A/B tests showing tangible gains in engagement and content diversity. The approach provides a foundational indexing mechanism for both discriminative and generative recommender systems in industrial-scale streaming platforms.

Abstract

Item indexing, which maps a large corpus of items into compact discrete representations, is critical for both discriminative and generative recommender systems, yet existing Vector Quantization (VQ)-based approaches struggle with the highly skewed and non-stationary item distributions common in streaming industry recommenders, leading to poor assignment accuracy, imbalanced cluster occupancy, and insufficient cluster separation. To address these challenges, we propose MERGE, a next-generation item indexing paradigm that adaptively constructs clusters from scratch, dynamically monitors cluster occupancy, and forms hierarchical index structures via fine-to-coarse merging. Extensive experiments demonstrate that MERGE significantly improves assignment accuracy, cluster uniformity, and cluster separation compared with existing indexing methods, while online A/B tests show substantial gains in key business metrics, highlighting its potential as a foundational indexing approach for large-scale recommendation.

Figures (6)

• Figure 1: Overview of the MERGE framework. The MERGE framework processes item batches by matching them to the codebook using cosine similarity. Successfully matched items update existing clusters, while unmatched items form new clusters via union-find. Valid new clusters are either appended or filled, and the rest are recycled. After building the fine codebook, a merging process creates a coarse codebook, forming a hierarchical structure for efficient indexing.
• Figure 2: Accuracy comparison between MERGE and VQ, illustrating the item-to-cluster cosine similarity (I2C CosSim) distributions within the codebook.
• Figure 3: Uniformity comparison between MERGE and VQ, showing cluster sizes and the cumulative distribution of videos within the VV bucket range across clusters. This indicates that, for a given VV bucket, the total area corresponding to each color sums to 1.
• Figure 4: Separation comparison between MERGE and VQ, illustrating the cluster-to-cluster cosine similarity (C2C CosSim) distributions within the codebook.
• Figure 5: Overview of MERGE deployment during serving, following the Trinity Trinity pipeline. MERGE provides a bidirectional mapping between clusters and items.