Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search

Yingchen Zhang, Ruqing Zhang, Jiafeng Guo, Wenjun Peng, Sen Li, Fuyu Lv, Xueqi Cheng

TL;DR

The work addresses the challenge of designing semantically rich yet search-space–constrained document identifiers for generative retrieval. It introduces C2T-ID, a method that converts semantic codebooks into textual docids by building a hierarchical clustering tree, extracting per-node keywords, and replacing numeric labels with top-$K$ keywords, with an optional semantic smoothing step for fluency. This approach preserves hierarchical decoding constraints while providing richer semantic priors to large language models, enabling more effective document retrieval. Experiments on Natural Questions and Taobao product search show that C2T-ID outperforms atomic, semantic codebook, and pure-text docids, with particularly strong gains on large-scale, complex tasks and robust zero-shot performance.

Abstract

Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints.

C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search

TL;DR

The work addresses the challenge of designing semantically rich yet search-space–constrained document identifiers for generative retrieval. It introduces C2T-ID, a method that converts semantic codebooks into textual docids by building a hierarchical clustering tree, extracting per-node keywords, and replacing numeric labels with top- keywords, with an optional semantic smoothing step for fluency. This approach preserves hierarchical decoding constraints while providing richer semantic priors to large language models, enabling more effective document retrieval. Experiments on Natural Questions and Taobao product search show that C2T-ID outperforms atomic, semantic codebook, and pure-text docids, with particularly strong gains on large-scale, complex tasks and robust zero-shot performance.

Abstract

Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints.
Paper Structure (13 sections, 1 equation, 2 figures, 2 tables)

This paper contains 13 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related works on Genetative Retrieval (GR)
  3. Method
  4. Overview
  5. Semantic codebooks construction
  6. Document prior information extraction
  7. Textual label replacement
  8. Semantic smoothing (Optional)
  9. Experiments
  10. Experimental settings
  11. Experimental results
  12. Conclusion and limitations
  13. Acknowledgments

Figures (2)

  • Figure 1: Comparison of C2T-ID with other docid methods. (a) is an example of the decoding for semantic codebook docid, (b) for textual docid, and (c) for our C2T-ID.
  • Figure 2: Results of two-level semantic smoothing experiment on the NQ dataset.