Doc2Token: Bridging Vocabulary Gap by Predicting Missing Tokens for E-commerce Search
Kaihao Li, Juexin Lin, Tony Lee
TL;DR
The paper tackles vocabulary mismatch in e-commerce search by introducing Doc2Token, a token-level document expansion method that predicts novel tokens missing from product metadata. It defines a novel ROUGE (nROUGE) metric to assess novelty and shows that Doc2Token achieves higher nROUGE and token diversity than Doc2Query, while also improving efficiency. The approach uses a frequency-weighted seq2seq model based on T5 and beam-search decoding, with production deployment in Walmart.com’s search pipeline yielding measurable revenue gains. Overall, Doc2Token provides a practical, interpretable mechanism to bridge lexical gaps in product search, enhancing retrieval and monetization without relying solely on dense representations.
Abstract
Addressing the "vocabulary mismatch" issue in information retrieval is a central challenge for e-commerce search engines, because product pages often miss important keywords that customers search for. Doc2Query[1] is a popular document-expansion technique that predicts search queries for a document and includes the predicted queries with the document for retrieval. However, this approach can be inefficient for e-commerce search, because the predicted query tokens are often already present in the document. In this paper, we propose Doc2Token, a technique that predicts relevant tokens (instead of queries) that are missing from the document and includes these tokens in the document for retrieval. For the task of predicting missing tokens, we introduce a new metric, "novel ROUGE score". Doc2Token is demonstrated to be superior to Doc2Query in terms of novel ROUGE score and diversity of predictions. Doc2Token also exhibits efficiency gains by reducing both training and inference times. We deployed the feature to production and observed significant revenue gain in an online A/B test, and launched the feature to full traffic on Walmart.com. [1] R. Nogueira, W. Yang, J. Lin, K. Cho, Document expansion by query prediction, arXiv preprint arXiv:1904.08375 (2019)