Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generation

Xinping Zhao, Dongfang Li, Yan Zhong, Boren Hu, Yibin Chen, Baotian Hu, Min Zhang

TL;DR

A model-based evidence extraction learning framework, SEER, optimizing a vanilla model as an evidence extractor with desired properties through self-aligned learning is proposed, which largely improves the final RAG performance, enhances the faithfulness, helpfulness, and conciseness of the extracted evidence, and reduces the evidence length by 9.25 times.

Abstract

Recent studies in Retrieval-Augmented Generation (RAG) have investigated extracting evidence from retrieved passages to reduce computational costs and enhance the final RAG performance, yet it remains challenging. Existing methods heavily rely on heuristic-based augmentation, encountering several issues: (1) Poor generalization due to hand-crafted context filtering; (2) Semantics deficiency due to rule-based context chunking; (3) Skewed length due to sentence-wise filter learning. To address these issues, we propose a model-based evidence extraction learning framework, SEER, optimizing a vanilla model as an evidence extractor with desired properties through self-aligned learning. Extensive experiments show that our method largely improves the final RAG performance, enhances the faithfulness, helpfulness, and conciseness of the extracted evidence, and reduces the evidence length by 9.25 times. The code will be available at https://github.com/HITsz-TMG/SEER.

SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generation

TL;DR

A model-based evidence extraction learning framework, SEER, optimizing a vanilla model as an evidence extractor with desired properties through self-aligned learning is proposed, which largely improves the final RAG performance, enhances the faithfulness, helpfulness, and conciseness of the extracted evidence, and reduces the evidence length by 9.25 times.

Abstract

Recent studies in Retrieval-Augmented Generation (RAG) have investigated extracting evidence from retrieved passages to reduce computational costs and enhance the final RAG performance, yet it remains challenging. Existing methods heavily rely on heuristic-based augmentation, encountering several issues: (1) Poor generalization due to hand-crafted context filtering; (2) Semantics deficiency due to rule-based context chunking; (3) Skewed length due to sentence-wise filter learning. To address these issues, we propose a model-based evidence extraction learning framework, SEER, optimizing a vanilla model as an evidence extractor with desired properties through self-aligned learning. Extensive experiments show that our method largely improves the final RAG performance, enhances the faithfulness, helpfulness, and conciseness of the extracted evidence, and reduces the evidence length by 9.25 times. The code will be available at https://github.com/HITsz-TMG/SEER.

Paper Structure

This paper contains 22 sections, 8 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Augmentation Analysis
  5. Methodology
  6. Evidence Extraction Stage
  7. Expert Assessment Stage
  8. Self-Alignment Stage
  9. Experiments
  10. Experimental Settings
  11. Model Comparison (RQ1)
  12. Alignment Study (RQ2)
  13. Robustness Analysis (RQ3)
  14. Ablation Study (RQ4)
  15. Related works
  16. ...and 7 more sections

Figures (6)

  • Figure 1: The RAG pipeline with the evidence extractor, in which the supporting content and the distracting content are marked in green and yellow, respectively.
  • Figure 2: Comparison between model-based and heuristic-based augmentation w.r.t. context relevance.
  • Figure 3: The overall system framework of our SEER, which mainly consists of three modeling stages.
  • Figure 4: Alignment performance w.r.t. faithfulness, helpfulness, and conciseness. The bar represents the oracle scores, while the line denotes the percentage of performance improvement in comparison with the Base.
  • Figure 5: Model performance w.r.t. Noise-to-Signal Ratio (NSR) ratio. The bar denotes the silver faithfulness score or the helpfulness score, while the line represents the performance drop percent compared to the model that is provided with only relevant retrieved passages.
  • ...and 1 more figures