Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Fact Retrieval in PLMs through Truthfulness

Paul Youssef, Jörg Schlötterer, Christin Seifert

TL;DR

The use of a helper model, which assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs, is investigated to improve fact retrieval.

Abstract

Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.

Enhancing Fact Retrieval in PLMs through Truthfulness

TL;DR

The use of a helper model, which assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs, is investigated to improve fact retrieval.

Abstract

Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. There has a been an interest in quantifying and improving the amount of facts that can be extracted from PLMs, as they have been envisioned to act as soft knowledge bases, which can be queried in natural language. Different approaches exist to enhance fact retrieval from PLM. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. Leveraging this finding to improve factual knowledge retrieval remains unexplored. In this work, we investigate the use of a helper model to improve fact retrieval. The helper model assesses the truthfulness of an input based on the corresponding hidden states representations from the PLMs. We evaluate this approach on several masked PLMs and show that it enhances fact retrieval by up to 33\%. Our findings highlight the potential of hidden states representations from PLMs in improving their factual knowledge retrieval.

Paper Structure

This paper contains 14 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fact retrieval.
  4. Truthfulness in Language Models.
  5. Methodology
  6. Experiments
  7. Datasets and PLMs
  8. Datasets.
  9. Training the helper model.
  10. PLMs.
  11. Results & Discussions
  12. Effect of Neg. Index ($k+1$)
  13. Conclusion
  14. Limitations

Figures (2)

  • Figure 1: An overview of our method for factual knowledge retrieval. A helper model decides which of the proposed answers is correct based on hidden state representations of the answer from the probed PLM.
  • Figure 2: Relation between Neg. Index (K+1) and Gain in Accuracy. Top: BERT-models, bottom: T5-models