Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction

TL;DR

This paper proposes a novel approach for few-shot relation extraction using large language models, named CoT-ER, chain-of-thought with explicit evidence reasoning, which first induces largelanguage models to generate evidences using task-specific and concept-level knowledge and is explicitly incorporated into chain- of-thought prompting for relation extraction.

Abstract

Few-shot relation extraction involves identifying the type of relationship between two specific entities within a text, using a limited number of annotated samples. A variety of solutions to this problem have emerged by applying meta-learning and neural graph techniques which typically necessitate a training process for adaptation. Recently, the strategy of in-context learning has been demonstrating notable results without the need of training. Few studies have already utilized in-context learning for zero-shot information extraction. Unfortunately, the evidence for inference is either not considered or implicitly modeled during the construction of chain-of-thought prompts. In this paper, we propose a novel approach for few-shot relation extraction using large language models, named CoT-ER, chain-of-thought with explicit evidence reasoning. In particular, CoT-ER first induces large language models to generate evidences using task-specific and concept-level knowledge. Then these evidences are explicitly incorporated into chain-of-thought prompting for relation extraction. Experimental results demonstrate that our CoT-ER approach (with 0% training data) achieves competitive performance compared to the fully-supervised (with 100% training data) state-of-the-art approach on the FewRel1.0 and FewRel2.0 datasets.

Figures (4)

• Figure 1: The comparison between Auto-CoT and CoT-ER (ours) prompting methods. Specifically, CoT-ER leverages side information to induce LLMs to generate explicit evidence for relation reasoning.
• Figure 2: An illustration of CoT-ER for few-shot RE. Different colored lines indicate the flow of support and query instances from an N-way K-shot task. a) Human-instructed reasoning module (§\ref{['sec:human-instructed-reasoning-module']}) associates an evidence reasoning process with each instance from the support set by prompting LLM with human-annotated demonstrations; b) Instances retrieval module (§\ref{['sec:retrieval-module']}) selects the few-shot demonstrations from the candidate set for the ultimate prompt based on their similarity to the query instance. c) Inference module (§\ref{['sec:inference-module']}) utilizes the ultimate prompt, which is composed of $\mathcal{M}$ support instances with their associated reasoning process, to derive an evidence reasoning process for the query instance.
• Figure 3: Template of the ultimate prompt. $\mathcal{M}$ represents the number of few-shot demonstrations selected by the instance retrieval module, and $Verbalize(·)$ denotes a transformation function that combines the component into a coherent expression. The prompt used in human human-instructed reasoning module follows a similar structure, but instead of few-shot demonstrations, it employs annotated examples.
• Figure 4: Ablation study on the first and second reasoning steps of CoT-ER. Auto-CoT refers to the "with reasoning generation" version.