Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval

Weihang Su, Qingyao Ai, Xiangsheng Li, Jia Chen, Yiqun Liu, Xiaolong Wu, Shengluan Hou

TL;DR

This work tackles ad-hoc information retrieval by leveraging Wikipedia's rich structured information rather than relying solely on plain text. It introduces Wikiformer, a pre-trained model with four objectives—Simulated Re-ranking, Representative Words Identification, Abstract Texts Identification, and Long Texts Matching—that sample pseudo query-document pairs from Wikipedia's titles, abstracts, headings, and See Also links. Empirical results across MS MARCO, TREC DL 2019, TREC Covid, LeCaRD, and CAIL-LCR demonstrate strong zero-shot and fine-tuned performance, including in vertical domains and long-text matching tasks. Ablation analyses confirm that each objective contributes to gains, and the approach shows data-efficient pre-training on structured knowledge.

Abstract

With the development of deep learning and natural language processing techniques, pre-trained language models have been widely used to solve information retrieval (IR) problems. Benefiting from the pre-training and fine-tuning paradigm, these models achieve state-of-the-art performance. In previous works, plain texts in Wikipedia have been widely used in the pre-training stage. However, the rich structured information in Wikipedia, such as the titles, abstracts, hierarchical heading (multi-level title) structure, relationship between articles, references, hyperlink structures, and the writing organizations, has not been fully explored. In this paper, we devise four pre-training objectives tailored for IR tasks based on the structured knowledge of Wikipedia. Compared to existing pre-training methods, our approach can better capture the semantic knowledge in the training corpus by leveraging the human-edited structured data from Wikipedia. Experimental results on multiple IR benchmark datasets show the superior performance of our model in both zero-shot and fine-tuning settings compared to existing strong retrieval baselines. Besides, experimental results in biomedical and legal domains demonstrate that our approach achieves better performance in vertical domains compared to previous models, especially in scenarios where long text similarity matching is needed.

Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval

TL;DR

This work tackles ad-hoc information retrieval by leveraging Wikipedia's rich structured information rather than relying solely on plain text. It introduces Wikiformer, a pre-trained model with four objectives—Simulated Re-ranking, Representative Words Identification, Abstract Texts Identification, and Long Texts Matching—that sample pseudo query-document pairs from Wikipedia's titles, abstracts, headings, and See Also links. Empirical results across MS MARCO, TREC DL 2019, TREC Covid, LeCaRD, and CAIL-LCR demonstrate strong zero-shot and fine-tuned performance, including in vertical domains and long-text matching tasks. Ablation analyses confirm that each objective contributes to gains, and the approach shows data-efficient pre-training on structured knowledge.

Abstract

With the development of deep learning and natural language processing techniques, pre-trained language models have been widely used to solve information retrieval (IR) problems. Benefiting from the pre-training and fine-tuning paradigm, these models achieve state-of-the-art performance. In previous works, plain texts in Wikipedia have been widely used in the pre-training stage. However, the rich structured information in Wikipedia, such as the titles, abstracts, hierarchical heading (multi-level title) structure, relationship between articles, references, hyperlink structures, and the writing organizations, has not been fully explored. In this paper, we devise four pre-training objectives tailored for IR tasks based on the structured knowledge of Wikipedia. Compared to existing pre-training methods, our approach can better capture the semantic knowledge in the training corpus by leveraging the human-edited structured data from Wikipedia. Experimental results on multiple IR benchmark datasets show the superior performance of our model in both zero-shot and fine-tuning settings compared to existing strong retrieval baselines. Besides, experimental results in biomedical and legal domains demonstrate that our approach achieves better performance in vertical domains compared to previous models, especially in scenarios where long text similarity matching is needed.
Paper Structure (23 sections, 7 equations, 4 figures, 5 tables)

This paper contains 23 sections, 7 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pre-trained Language Models
  4. Pre-training Methods Tailored for IR
  5. Methodology
  6. Simulated Re-ranking (SRR)
  7. Representative Words Identification (RWI)
  8. Abstract Texts Identification (ATI)
  9. Long Texts Matching (LTM)
  10. Experiments
  11. Dataset Description
  12. Baselines for Comparison
  13. Implementation Details
  14. Evaluation Methodology
  15. Experimental Results
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Rich structured information of Wikipedia.
  • Figure 2: Pseudo query-document pairs generated from the tree structure of a Wikipedia article, where $q$ is the query, $d^+$ is the positive document and $d^-$ are negative documents.
  • Figure 3: The contrastive sampling strategy of the RWI task, where $D$ is the document, $q+$ is the positive query and $q^-$ is the negative query.
  • Figure 4: The performance of Wikiformer at different sizes of pre-training data sampled from the SRR task. The red dotted line shows the performance of Query Likelihood.