CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks

TL;DR

The paper tackles the problem of dynamic retrieval in knowledge-intensive language tasks by introducing continual document learning (CDL) and the KILT++ benchmark, revealing that CorpusBrain struggles with catastrophic forgetting when the corpus evolves. It proposes CorpusBrain++, a backbone-adapter-based continual pre-training framework with task-specific adapters, dedicated pre-training objectives, and an experience-replay mechanism to learn from both new and revisited old documents. Empirical results show that CorpusBrain++ consistently outperforms traditional and generative IR baselines in dynamic CDL scenarios while maintaining efficiency, and they demonstrate strong forward transfer and reduced forgetting. The work paves the way for up-to-date, knowledge-grounded dialogue and search systems that can adapt to growing knowledge sources in real time.

Abstract

Knowledge-intensive language tasks (KILTs) typically require retrieving relevant documents from trustworthy corpora, e.g., Wikipedia, to produce specific answers. Very recently, a pre-trained generative retrieval model for KILTs, named CorpusBrain, was proposed and reached new state-of-the-art retrieval performance. However, most existing research on KILTs, including CorpusBrain, has predominantly focused on a static document collection, overlooking the dynamic nature of real-world scenarios, where new documents are continuously being incorporated into the source corpus. To address this gap, it is crucial to explore the capability of retrieval models to effectively handle the dynamic retrieval scenario inherent in KILTs. In this work, we first introduce the continual document learning (CDL) task for KILTs and build a novel benchmark dataset named KILT++ based on the original KILT dataset for evaluation. Then, we conduct a comprehensive study over the use of pre-trained CorpusBrain on KILT++. Unlike the promising results in the stationary scenario, CorpusBrain is prone to catastrophic forgetting in the dynamic scenario, hence hampering the retrieval performance. To alleviate this issue, we propose CorpusBrain++, a continual generative pre-training framework. Empirical results demonstrate the significant effectiveness and remarkable efficiency of CorpusBrain++ in comparison to both traditional and generative IR methods.

Figures (8)

• Figure 1: Comparison of CorpusBrain and CorpusBrain++. CorpusBrain can solely support one-time document learning and service deployment, without the ability to assimilate new documents and dynamically update the knowledge base. Beyond CorpusBrain, the dynamic CorpusBrain++ can support continual document learning and service deployment to adapt to evolving corpus in the realistic scenario.
• Figure 2: Evaluation criteria of the continual document learning task for KILTs.
• Figure 3: Illustration of our proposed CorpusBrain++ method. The backbone first involves pre-training on the initial base document set $D_0$ through the ISS, LPS, and HIP pretraining tasks, and fine-tuning with golden pairs derived from the KILT++ training set. To accommodate each task, a specific adapter is allocated, and a dedicated pretraining task is introduced to mimic the characteristics of downstream input queries. In addition to the incremental document set $D_t$, we also revisit semantically similar documents to $D_t$ from previous sessions, thereby generating pseudo pairs and continually pre-training the adapters.
• Figure 4: The architecture of CorpusBrain and CorpusBrain++.
• Figure 5: Illustration of specific pre-training tasks for each KILT task. Anchor texts are marked in blue. The colored underlines in Wikipedia content correspond to the source text of the corresponding KILT task in the table below. Query examples provide an example of the corresponding downstream KILT task. Input and output refer to the constructed input pseudo-queries and corresponding output docids.