Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts

Yuri Facanha Bezerra, Li Weigang

TL;DR

LLMQuoter tackles the inefficiency of reasoning over large contexts in Retrieval-Augmented Generation by introducing a quote-first-then-answer pipeline. It trains a lightweight student on a distillation dataset generated from a high-performing teacher, fine-tuning a LLaMA-3B model with LoRA to extract the most relevant quotes from context $C$ in response to a question $Q$, and then passes these quotes to a reasoning module. In experiments on a $15{,}000$-sample subset of HotpotQA, LLMQuoter outperformed full-context baselines such as RAFT, achieving over $20$-point accuracy gains while reducing computation. The work demonstrates that distilled, quote-based reasoning can democratize access to effective RAG capabilities and offers a scalable framework adaptable to diverse domains.

Abstract

We introduce LLMQuoter, a lightweight, distillation-based model designed to enhance Retrieval Augmented Generation (RAG) by extracting the most relevant textual evidence for downstream reasoning tasks. Built on the LLaMA-3B architecture and fine-tuned with Low-Rank Adaptation (LoRA) on a 15,000-sample subset of HotpotQA, LLMQuoter adopts a "quote-first-then-answer" strategy, efficiently identifying key quotes before passing curated snippets to reasoning models. This workflow reduces cognitive overhead and outperforms full-context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language models. By leveraging knowledge distillation from a high-performing teacher model, LLMQuoter achieves competitive results in a resource-efficient fine-tuning setup. It democratizes advanced RAG capabilities, delivering significant performance improvements without requiring extensive model retraining. Our results highlight the potential of distilled quote-based reasoning to streamline complex workflows, offering a scalable and practical solution for researchers and practitioners alike.

LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts

TL;DR

LLMQuoter tackles the inefficiency of reasoning over large contexts in Retrieval-Augmented Generation by introducing a quote-first-then-answer pipeline. It trains a lightweight student on a distillation dataset generated from a high-performing teacher, fine-tuning a LLaMA-3B model with LoRA to extract the most relevant quotes from context in response to a question , and then passes these quotes to a reasoning module. In experiments on a -sample subset of HotpotQA, LLMQuoter outperformed full-context baselines such as RAFT, achieving over -point accuracy gains while reducing computation. The work demonstrates that distilled, quote-based reasoning can democratize access to effective RAG capabilities and offers a scalable framework adaptable to diverse domains.

Abstract

We introduce LLMQuoter, a lightweight, distillation-based model designed to enhance Retrieval Augmented Generation (RAG) by extracting the most relevant textual evidence for downstream reasoning tasks. Built on the LLaMA-3B architecture and fine-tuned with Low-Rank Adaptation (LoRA) on a 15,000-sample subset of HotpotQA, LLMQuoter adopts a "quote-first-then-answer" strategy, efficiently identifying key quotes before passing curated snippets to reasoning models. This workflow reduces cognitive overhead and outperforms full-context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language models. By leveraging knowledge distillation from a high-performing teacher model, LLMQuoter achieves competitive results in a resource-efficient fine-tuning setup. It democratizes advanced RAG capabilities, delivering significant performance improvements without requiring extensive model retraining. Our results highlight the potential of distilled quote-based reasoning to streamline complex workflows, offering a scalable and practical solution for researchers and practitioners alike.
Paper Structure (14 sections, 7 equations, 2 figures)

This paper contains 14 sections, 7 equations, 2 figures.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. LLM Reasoning
  4. Knowledge Distillation in LLMs
  5. RAFT (RAG + FT)
  6. Semantic Evaluation
  7. METHODOLOGY
  8. Problem Formalization
  9. LLM Distillation
  10. Fine-Tuning LLM with LoRA
  11. Evaluation Framework and Metrics
  12. DSpy-Assisted Validation with LLM Judge:
  13. Proving the Benefit of Using Quotes
  14. EXPERIMENTS

Figures (2)

  • Figure 1: RAFT inference exampleraft
  • Figure :