Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models

Stephan Linzbach, Dimitar Dimitrov, Laura Kallmeyer, Kilian Evang, Hajira Jabeen, Stefan Dietze

TL;DR

The paper addresses how prompt syntax and supplementary information shape knowledge retrieval from pretrained language models by introducing ConPare-LAMA, a large, controlled paraphrasing probe. It uses a unified meta-template to separate syntactic form (clausal vs appositive) from semantic content (domain/range information) across 60 relations and three corpora, evaluating base models BERT, RoBERTa, and Luke. Key findings show clausal syntax improves performance, consistency, and reduces uncertainty, with range information providing larger gains than domain information, while appositive syntax tends to add noise and lower reliability. These insights guide syntax-aware prompt design and suggest directions for syntax-conscious pre-training to enhance robust knowledge retrieval in PLMs.

Abstract

Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects or objects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms.

Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models

TL;DR

The paper addresses how prompt syntax and supplementary information shape knowledge retrieval from pretrained language models by introducing ConPare-LAMA, a large, controlled paraphrasing probe. It uses a unified meta-template to separate syntactic form (clausal vs appositive) from semantic content (domain/range information) across 60 relations and three corpora, evaluating base models BERT, RoBERTa, and Luke. Key findings show clausal syntax improves performance, consistency, and reduces uncertainty, with range information providing larger gains than domain information, while appositive syntax tends to add noise and lower reliability. These insights guide syntax-aware prompt design and suggest directions for syntax-conscious pre-training to enhance robust knowledge retrieval in PLMs.

Abstract

Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects or objects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms.
Paper Structure (13 sections, 5 figures, 3 tables)

This paper contains 13 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controlled Paraphrasing
  4. Dissecting Paraphrasing
  5. Meta-Template Definition
  6. Template Instantiation
  7. ConPare-LAMA
  8. Experiments
  9. Results
  10. Discussion
  11. Conclusion & Future Work
  12. Limitations
  13. Ethical Considerations

Figures (5)

  • Figure 1: Relationship between prompt types (simple (orange), compound (blue), complex (red), compound-complex (purple)), syntactic forms (clausal and appositive), and sInf combinations (relation, relation+range or domain, relation+combined) used to study the influence of syntax on knowledge retrieval.
  • Figure 2: Meta-template that facilitates comparable prompt creation for various relations and information demands.
  • Figure 3: Knowledge consistency for sInf added through (a) clausal and (b) appositive prompts for all intersections of correctly predicted triples by RoBERTa on the TREx corpus.
  • Figure 4: Effect of combined domain+range information using either clausal (purple) or appositive (orange) syntax, compared to expected interval (black). Upper bound is choosing the better answer with either domain or range information, lower bound is choosing the one with the higher confidence.
  • Figure 5: Average binary entropy of response distribution of known subset (correct prediction in top 10) for the TREx corpora with differently completed prompts. Clausal syntax leaves less uncertainty. PLMs even generalize this loss in uncertainty to the combined setting given clausal syntax.