PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval

TL;DR

The paper addresses the challenge of scaling dense retrieval to streaming, multilingual, terabyte-scale corpora where static PLAID representations degrade as documents arrive over time. It proposes PLAID SHIRTTT (FLOOBY), a hierarchical sharding and multi-phase incremental indexing architecture that enables efficient, streaming-compatible indexing for a ColBERT variant. Experiments on ClueWeb09 and the multilingual NeuCLIR collection demonstrate effectiveness at unprecedented scale for ColBERT architectures and confirm the approach's applicability to multilingual settings. The work contributes an architectural solution, empirical validation on large and multilingual datasets, and marks the first application of a ColBERT variant to terabyte-scale collections.

Abstract

PLAID, an efficient implementation of the ColBERT late interaction bi-encoder using pretrained language models for ranking, consistently achieves state-of-the-art performance in monolingual, cross-language, and multilingual retrieval. PLAID differs from ColBERT by assigning terms to clusters and representing those terms as cluster centroids plus compressed residual vectors. While PLAID is effective in batch experiments, its performance degrades in streaming settings where documents arrive over time because representations of new tokens may be poorly modeled by the earlier tokens used to select cluster centroids. PLAID Streaming Hierarchical Indexing that Runs on Terabytes of Temporal Text (PLAID SHIRTTT) addresses this concern using multi-phase incremental indexing based on hierarchical sharding. Experiments on ClueWeb09 and the multilingual NeuCLIR collection demonstrate the effectiveness of this approach both for the largest collection indexed to date by the ColBERT architecture and in the multilingual setting, respectively.