Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Document Refinement for Long-context Retrieval-augmented Generation

Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Qi Ye, Zhicheng Dou

TL;DR

LongRefiner tackles the challenge of long-context retrieval-augmented generation by introducing a document-level refiner that leverages dual-level query analysis, hierarchical document structuring, and adaptive refinement. It combines an XML-based, compact representation of document structure with a offline-online inference scheme and multi-task LoRA learning on a single backbone, enabling efficient refinement with reduced token budgets. Across seven QA datasets, it achieves competitive or superior generation quality while reducing token usage by approximately 10x and online latency by about 75% relative to full-content baselines. The approach offers practical, scalable design patterns for long-text RAG deployments and is complemented by an open-source code release.

Abstract

Real-world RAG applications often encounter long-context input scenarios, where redundant information and noise results in higher inference costs and reduced performance. To address these challenges, we propose LongRefiner, an efficient plug-and-play refiner that leverages the inherent structural characteristics of long documents. LongRefiner employs dual-level query analysis, hierarchical document structuring, and adaptive refinement through multi-task learning on a single foundation model. Experiments on seven QA datasets demonstrate that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to the best baseline. Further analysis validates that LongRefiner is scalable, efficient, and effective, providing practical insights for real-world long-text RAG applications. Our code is available at https://github.com/ignorejjj/LongRefiner.

Hierarchical Document Refinement for Long-context Retrieval-augmented Generation

TL;DR

LongRefiner tackles the challenge of long-context retrieval-augmented generation by introducing a document-level refiner that leverages dual-level query analysis, hierarchical document structuring, and adaptive refinement. It combines an XML-based, compact representation of document structure with a offline-online inference scheme and multi-task LoRA learning on a single backbone, enabling efficient refinement with reduced token budgets. Across seven QA datasets, it achieves competitive or superior generation quality while reducing token usage by approximately 10x and online latency by about 75% relative to full-content baselines. The approach offers practical, scalable design patterns for long-text RAG deployments and is complemented by an open-source code release.

Abstract

Real-world RAG applications often encounter long-context input scenarios, where redundant information and noise results in higher inference costs and reduced performance. To address these challenges, we propose LongRefiner, an efficient plug-and-play refiner that leverages the inherent structural characteristics of long documents. LongRefiner employs dual-level query analysis, hierarchical document structuring, and adaptive refinement through multi-task learning on a single foundation model. Experiments on seven QA datasets demonstrate that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to the best baseline. Further analysis validates that LongRefiner is scalable, efficient, and effective, providing practical insights for real-world long-text RAG applications. Our code is available at https://github.com/ignorejjj/LongRefiner.
Paper Structure (47 sections, 7 equations, 4 figures, 6 tables)

This paper contains 47 sections, 7 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. LongRefiner: a Long Document Refiner for RAG
  4. Dual-Level Query Analysis
  5. Hierarchical Document Structuring
  6. Tree-based structured document definition.
  7. XML-format document structure representation.
  8. (1) Structure generation.
  9. (2) Content filling based on structure
  10. Wikipedia-Based Label Collection
  11. Adaptive Document Refinement
  12. Local Perspective.
  13. Global Perspective.
  14. Combination.
  15. Training and Inference
  16. ...and 32 more sections

Figures (4)

  • Figure 1: Comparison of different methods in terms of efficiency and effectiveness.
  • Figure 2: Overview of the LongRefiner Framework.
  • Figure 3: Analysis of scaling the base model size (left) and training data amount (right) in hierarchical document structuring step on TriviaQA. Recall represents the proportion of golden answer in the input prompt.
  • Figure 4: The performance of LongRefiner across different document lengths, where Doc Length refers to the total number of tokens for all retrieved documents corresponding to a query.