Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

Qian Dong, Yiding Liu, Suqi Cheng, Shuaiqiang Wang, Zhicong Cheng, Shuzi Niu, Dawei Yin

TL;DR

The paper tackles the limitation of PLM-based passage re-ranking in capturing domain-specific and heterogeneous knowledge. It introduces KERM, a knowledge-enhanced reranker that first distills a noisy external knowledge graph into a reliable knowledge meta-graph and then uses a Graph Meta Network to propagate and align explicit knowledge with implicit textual representations via a novel knowledge injector. The framework combines a text encoder with multi-hop KG reasoning to improve query–passage relevance, trained with MLM and SRP pre-training and fine-tuned with cross-entropy on ranking data. Empirical results on MSMARCO-DEV, TREC 2019 DL, and Ohsumed show state-of-the-art or competitive performance, with ablations confirming the importance of global KG pruning, local meta-graphs, and the knowledge injector. The work advances practical knowledge-enabled re-ranking, particularly in domains with limited training data, and points to scalable extensions for retrieval-time usage and end-to-end KG generation.

Abstract

Passage re-ranking is to obtain a permutation over the candidate passage set from retrieval stage. Re-rankers have been boomed by Pre-trained Language Models (PLMs) due to their overwhelming advantages in natural language understanding. However, existing PLM based re-rankers may easily suffer from vocabulary mismatch and lack of domain specific knowledge. To alleviate these problems, explicit knowledge contained in knowledge graph is carefully introduced in our work. Specifically, we employ the existing knowledge graph which is incomplete and noisy, and first apply it in passage re-ranking task. To leverage a reliable knowledge, we propose a novel knowledge graph distillation method and obtain a knowledge meta graph as the bridge between query and passage. To align both kinds of embedding in the latent space, we employ PLM as text encoder and graph neural network over knowledge meta graph as knowledge encoder. Besides, a novel knowledge injector is designed for the dynamic interaction between text and knowledge encoder. Experimental results demonstrate the effectiveness of our method especially in queries requiring in-depth domain knowledge.

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

TL;DR

The paper tackles the limitation of PLM-based passage re-ranking in capturing domain-specific and heterogeneous knowledge. It introduces KERM, a knowledge-enhanced reranker that first distills a noisy external knowledge graph into a reliable knowledge meta-graph and then uses a Graph Meta Network to propagate and align explicit knowledge with implicit textual representations via a novel knowledge injector. The framework combines a text encoder with multi-hop KG reasoning to improve query–passage relevance, trained with MLM and SRP pre-training and fine-tuned with cross-entropy on ranking data. Empirical results on MSMARCO-DEV, TREC 2019 DL, and Ohsumed show state-of-the-art or competitive performance, with ablations confirming the importance of global KG pruning, local meta-graphs, and the knowledge injector. The work advances practical knowledge-enabled re-ranking, particularly in domains with limited training data, and points to scalable extensions for retrieval-time usage and end-to-end KG generation.

Abstract

Passage re-ranking is to obtain a permutation over the candidate passage set from retrieval stage. Re-rankers have been boomed by Pre-trained Language Models (PLMs) due to their overwhelming advantages in natural language understanding. However, existing PLM based re-rankers may easily suffer from vocabulary mismatch and lack of domain specific knowledge. To alleviate these problems, explicit knowledge contained in knowledge graph is carefully introduced in our work. Specifically, we employ the existing knowledge graph which is incomplete and noisy, and first apply it in passage re-ranking task. To leverage a reliable knowledge, we propose a novel knowledge graph distillation method and obtain a knowledge meta graph as the bridge between query and passage. To align both kinds of embedding in the latent space, we employ PLM as text encoder and graph neural network over knowledge meta graph as knowledge encoder. Besides, a novel knowledge injector is designed for the dynamic interaction between text and knowledge encoder. Experimental results demonstrate the effectiveness of our method especially in queries requiring in-depth domain knowledge.
Paper Structure (28 sections, 22 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 28 sections, 22 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. PLMs based Re-rankers
  4. PLMs based Retrievers
  5. Knowledge Enhanced Pre-trained Language Models (KE-PLMs)
  6. Graph Neural Network
  7. Problem Formulation
  8. Passage Re-ranking
  9. Explicit Knowledge Enhanced Passage Re-ranking
  10. KERM
  11. Knowledge Graph Distillation
  12. Step 1: Global Graph Pruning
  13. Step2: Meta-Graph Construction
  14. Knowledge Aggregation
  15. Text Encoder
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The workflow of KERM.
  • Figure 2: The illustration of global graph pruning.
  • Figure 3: The illustration of a meta-graph constructed between the target entities of a query-passage pair. Note that some entities do not appear in the meta-graph, as they might not have any path that reaches the entities on the other side.
  • Figure 4: The architecture of KERM.