Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval

Zhirui Kuai, Zuxu Chen, Huimu Wang, Mingming Li, Dadong Miao, Binbin Wang, Xusong Chen, Li Kuang, Yuxing Han, Jiaxing Wang, Guoyu Tang, Lin Liu, Songlin Wang, Jingwei Zhuo

TL;DR

This paper analyses and addresses the "Hourglass" phenomenon in RQ-SID by identifying data sparsity and long-tailed distribution as the primary causes, and proposes effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.

Abstract

Generative retrieval (GR) has emerged as a transformative paradigm in search and recommender systems, leveraging numeric-based identifier representations to enhance efficiency and generalization. Notably, methods like TIGER employing Residual Quantization-based Semantic Identifiers (RQ-SID), have shown significant promise in e-commerce scenarios by effectively managing item IDs. However, a critical issue termed the "\textbf{Hourglass}" phenomenon, occurs in RQ-SID, where intermediate codebook tokens become overly concentrated, hindering the full utilization of generative retrieval methods. This paper analyses and addresses this problem by identifying data sparsity and long-tailed distribution as the primary causes. Through comprehensive experiments and detailed ablation studies, we analyze the impact of these factors on codebook utilization and data distribution. Our findings reveal that the "Hourglass" phenomenon substantially impacts the performance of RQ-SID in generative retrieval. We propose effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.

Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval

TL;DR

This paper analyses and addresses the "Hourglass" phenomenon in RQ-SID by identifying data sparsity and long-tailed distribution as the primary causes, and proposes effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.

Abstract

Generative retrieval (GR) has emerged as a transformative paradigm in search and recommender systems, leveraging numeric-based identifier representations to enhance efficiency and generalization. Notably, methods like TIGER employing Residual Quantization-based Semantic Identifiers (RQ-SID), have shown significant promise in e-commerce scenarios by effectively managing item IDs. However, a critical issue termed the "\textbf{Hourglass}" phenomenon, occurs in RQ-SID, where intermediate codebook tokens become overly concentrated, hindering the full utilization of generative retrieval methods. This paper analyses and addresses this problem by identifying data sparsity and long-tailed distribution as the primary causes. Through comprehensive experiments and detailed ablation studies, we analyze the impact of these factors on codebook utilization and data distribution. Our findings reveal that the "Hourglass" phenomenon substantially impacts the performance of RQ-SID in generative retrieval. We propose effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.
Paper Structure (16 sections, 4 equations, 6 figures, 2 tables)

This paper contains 16 sections, 4 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminary
  4. Residual Quantization
  5. Generative Retrieval
  6. Problem of GR based on RQ
  7. Hourglass Phenomenon
  8. Analysis of Residual Quantization
  9. Impact on the GR
  10. Methods and Experiments
  11. Heuristic Method
  12. Variable Length of SID
  13. Experiments
  14. Valid Ratio
  15. Conclusion
  16. ...and 1 more sections

Figures (6)

  • Figure 1: The Hourglass Phenomenon of Semantic IDs
  • Figure 2: Distribution and Connections of Semantic IDs
  • Figure 3: Illustrating the Hourglass Phenomenon in Semantic IDs with Different Statistical Metrics
  • Figure 4: Hierarchical Residual Reduction and Dimensional Analysis Across Layers
  • Figure 5: Invalid IDs Ratio when generating Semantic IDs using Beam Search for various values of $k$
  • ...and 1 more figures