Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Models Benefit from Preparation with Elicited Knowledge

Jiacan Yu, Hannah An, Lenhart K. Schubert

TL;DR

Zero-shot chain-of-thought prompts often underperform on QA tasks that hinge on factual knowledge. The paper proposes PREP, a dual-LM prompting approach where LM1 elicits relevant information and LM2 answers using that information, avoiding domain-specific prompt engineering. Empirical results on a curated artifact-part/material dataset and three commonsense benchmarks show PREP improves accuracy relative to baselines, particularly for medium-sized instruction-tuned LMs, and eliminates reliance on extensive hand-crafted prompts. This work offers a simple, training-free route to boost knowledge-heavy QA across diverse domains with minimal engineering effort.

Abstract

The zero-shot chain of thought (CoT) approach is often used in question answering (QA) by language models (LMs) for tasks that require multiple reasoning steps. However, some QA tasks hinge more on accessing relevant knowledge than on chaining reasoning steps. We introduce a simple prompting technique, called PREP, that involves using two instances of LMs: the first (LM1) generates relevant information, and the second (LM2) receives the information from the user and answers the question. This design is intended to make better use of the LM's instruction-following capability. PREP is applicable across various QA tasks without domain-specific prompt engineering. PREP is developed on a dataset of 100 QA questions, derived from an extensive schematic dataset specifying artifact parts and material composition. These questions ask which of two artifacts is less likely to share materials with another artifact. Such questions probe the LM's knowledge of shared materials in the part structure of different artifacts. We test our method on our parts-and-materials dataset and three published commonsense reasoning datasets. The average accuracy of our method is consistently higher than that of all the other tested methods across all the tested datasets.

Language Models Benefit from Preparation with Elicited Knowledge

TL;DR

Zero-shot chain-of-thought prompts often underperform on QA tasks that hinge on factual knowledge. The paper proposes PREP, a dual-LM prompting approach where LM1 elicits relevant information and LM2 answers using that information, avoiding domain-specific prompt engineering. Empirical results on a curated artifact-part/material dataset and three commonsense benchmarks show PREP improves accuracy relative to baselines, particularly for medium-sized instruction-tuned LMs, and eliminates reliance on extensive hand-crafted prompts. This work offers a simple, training-free route to boost knowledge-heavy QA across diverse domains with minimal engineering effort.

Abstract

The zero-shot chain of thought (CoT) approach is often used in question answering (QA) by language models (LMs) for tasks that require multiple reasoning steps. However, some QA tasks hinge more on accessing relevant knowledge than on chaining reasoning steps. We introduce a simple prompting technique, called PREP, that involves using two instances of LMs: the first (LM1) generates relevant information, and the second (LM2) receives the information from the user and answers the question. This design is intended to make better use of the LM's instruction-following capability. PREP is applicable across various QA tasks without domain-specific prompt engineering. PREP is developed on a dataset of 100 QA questions, derived from an extensive schematic dataset specifying artifact parts and material composition. These questions ask which of two artifacts is less likely to share materials with another artifact. Such questions probe the LM's knowledge of shared materials in the part structure of different artifacts. We test our method on our parts-and-materials dataset and three published commonsense reasoning datasets. The average accuracy of our method is consistently higher than that of all the other tested methods across all the tested datasets.
Paper Structure (25 sections, 3 figures, 7 tables)

This paper contains 25 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Related Work on Generic User-independent LM Prompting
  4. Prompting Methods that Require Task-Specific Design
  5. Our PREP Approach
  6. Experimental Settings
  7. Dataset Creation
  8. Experiments
  9. Results and Discussion
  10. Conclusions
  11. Model Details
  12. Prompt Details
  13. Sample Conversations
  14. Our PREP Method
  15. Direct
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Overview of our PREP approach, a dual-instance prompting method using general and user-knowledge independent prompts (slightly simplified here). The first prompt (P1), combined with the question, directs LM1 to provide specific facts in its response that seem relevant to answering the question. The collected information is then combined with the original question and used to prompt LM2.
  • Figure :
  • Figure :