Enhancing Relation Extraction via Supervised Rationale Verification and Feedback

TL;DR

This work addresses biased relation predictions in large language model–based relation extraction by introducing SRVF, a supervised rationale verification and feedback framework. SRVF collects unbiased and biased rationales via a causal intervention and observation pipeline, trains a rationale supervisor with contrastive learning to distinguish bias types, and, during inference, verifies generated rationales and retrieves targeted feedback demonstrations to iteratively correct predictions. Across SemEval, TACRED, and Re-TACRED—and also on document-level RE datasets DocRED and Re-DocRED—SRVF yields substantial improvements over in-context learning and established automated-feedback baselines, while maintaining competitive efficiency. The approach demonstrates strong generalization across LLM scales and backbones, reduces confusion between similar relations, and offers a practical, plug-in module for mitigating relation bias in RE tasks with potential applicability to other NLP tasks.

Abstract

Despite the rapid progress that existing automated feedback methods have made in correcting the output of large language models (LLMs), these methods cannot be well applied to the relation extraction (RE) task due to their designated feedback objectives and correction manner. To address this problem, we propose a novel automated feedback framework for RE, which presents a rationale supervisor to verify the rationale and provides re-selected demonstrations as feedback to correct the initial prediction. Specifically, we first design a causal intervention and observation method to collect biased/unbiased rationales for contrastive training the rationale supervisor. Then, we present a verification-feedback-correction procedure to iteratively enhance LLMs' capability of handling the RE task. Extensive experiments prove that our proposed framework significantly outperforms existing methods.

Figures (9)

• Figure 1: Comparison between current automated feedback methods (a) and ours (b). The main difference is that our rationale supervisor can verify whether the relation bias occurs and provide re-selected demonstrations as feedback.
• Figure 2: The structure causal model for illustrating the proposed causal intervention and observation strategy.
• Figure 3: An example of correcting the initial biased prediction of LLMs via the proposed SRVF framework in the inference time. The rationale supervisor first verifies the initial prediction in (a) as biased. Then, with the feedback demonstrations retrieved by the rationale supervisor, the LLM makes a correct relation prediction in (b). Note: The rationale supervisor here is obtained by contrastive training using collected biased and unbiased rationales as described before.
• Figure 4: Error matrix before and after the verification-feedback-correction procedure. The numbers show how many samples labeled $y$ (on the vertical axis) are incorrectly predicted as $x$ (on the horizontal axis).
• Figure 5: Efficiency comparison of different methods on the 5-shot SemEval setting. The results are accumulated along the X axis. For example, "After Initial Generation" refers to the sum time of "pre-inference" and "initial generation".