Inducing Diversity in Differentiable Search Indexing
Abhijeet Phatak, Jayant Sachdev, Sean D Rosario, Swati Kirti, Chittaranjan Tripathy
TL;DR
This work addresses the lack of diversity in Differentiable Search Index (DSI) retrieval by incorporating a diversity-enhancing objective into training. It introduces a loss that blends standard relevance with a diversity term inspired by Maximal Marginal Relevance (MMR), implemented as Loss_total = $\alpha \cdot \left(-\frac{1}{N} \sum_{i=1}^{N} y_i \log(p_i)\right) + (1 - \alpha) \cdot \sum_{d_i,d_j \in K, j>i} Sim_2(d_i, d_j)$, where top-K document representations are evaluated with cosine similarity. Evaluations on NQ320K and MSMARCO show that diversity can be induced during training with minimal impact on relevance while improving diversity metrics and without adding inference latency or post-processing steps. The results suggest the approach is beneficial for sub-topic retrieval and can be extended to incremental DSI settings for fast indexing updates. Overall, the paper advances end-to-end IR by enabling diverse recall directly within the neural index learning process, rather than relying on separate diversification stages.
Abstract
Differentiable Search Indexing (DSI) is a recent paradigm for information retrieval which uses a transformer-based neural network architecture as the document index to simplify the retrieval process. A differentiable index has many advantages enabling modifications, updates or extensions to the index. In this work, we explore balancing relevance and novel information content (diversity) for training DSI systems inspired by Maximal Marginal Relevance (MMR), and show the benefits of our approach over the naive DSI training. We present quantitative and qualitative evaluations of relevance and diversity measures obtained using our method on NQ320K and MSMARCO datasets in comparison to naive DSI. With our approach, it is possible to achieve diversity without any significant impact to relevance. Since we induce diversity while training DSI, the trained model has learned to diversify while being relevant. This obviates the need for a post-processing step to induce diversity in the recall set as typically performed using MMR. Our approach will be useful for Information Retrieval problems where both relevance and diversity are important such as in sub-topic retrieval. Our work can also be easily be extended to the incremental DSI settings which would enable fast updates to the index while retrieving a diverse recall set.