Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MDCure: A Scalable Pipeline for Multi-Document Instruction-Following

Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, Arman Cohan

TL;DR

MDCure presents a scalable, data-efficient framework for improving multi-document instruction-following in LLMs without extensive pre-training. It combines Generation of cross-document prompts with a fine-grained MDCureRM-based filtering step, and demonstrates compatibility with open- and proprietary models as well as policy optimization methods like PPO. Across 12K–72K synthetic MD instruction datasets, MDCure yields substantial gains on diverse MD and long-context benchmarks, with up to 75.1% average improvement and strong cross-domain generalization. The work highlights the value of MD-focused synthetic data and a cost-effective reward-based curation pipeline for enhancing MD reasoning while preserving general capabilities.

Abstract

Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present unique difficulties, including management of inter-document dependencies, redundancy, and incoherent structures. To address this challenge, we introduce MDCure, a scalable and effective instruction data generation framework to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human-annotated data. MDCure generates high-quality synthetic MD instruction data over sets of articles via targeted prompts. We also introduce MDCureRM, a cost-effective, MD-specific reward model to score and filter generated data based on their training utility for MD settings. MDCure is compatible with open- and closed-source models in addition to policy optimization methods such as PPO, enabling even small open-source models to surpass proprietary LLMs as strong generators of high-quality MD instruction data without further data filtering. With MDCure, we fine-tune a wide variety of LLMs up to 70B parameters in size from the FlanT5, Qwen2, and LLAMA3.1 model families. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks and domains show MDCure consistently improves performance over pre-trained baselines and base models by up to 75.1%. Our code, datasets, and models are available at https://github.com/yale-nlp/MDCure.

MDCure: A Scalable Pipeline for Multi-Document Instruction-Following

TL;DR

MDCure presents a scalable, data-efficient framework for improving multi-document instruction-following in LLMs without extensive pre-training. It combines Generation of cross-document prompts with a fine-grained MDCureRM-based filtering step, and demonstrates compatibility with open- and proprietary models as well as policy optimization methods like PPO. Across 12K–72K synthetic MD instruction datasets, MDCure yields substantial gains on diverse MD and long-context benchmarks, with up to 75.1% average improvement and strong cross-domain generalization. The work highlights the value of MD-focused synthetic data and a cost-effective reward-based curation pipeline for enhancing MD reasoning while preserving general capabilities.

Abstract

Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present unique difficulties, including management of inter-document dependencies, redundancy, and incoherent structures. To address this challenge, we introduce MDCure, a scalable and effective instruction data generation framework to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human-annotated data. MDCure generates high-quality synthetic MD instruction data over sets of articles via targeted prompts. We also introduce MDCureRM, a cost-effective, MD-specific reward model to score and filter generated data based on their training utility for MD settings. MDCure is compatible with open- and closed-source models in addition to policy optimization methods such as PPO, enabling even small open-source models to surpass proprietary LLMs as strong generators of high-quality MD instruction data without further data filtering. With MDCure, we fine-tune a wide variety of LLMs up to 70B parameters in size from the FlanT5, Qwen2, and LLAMA3.1 model families. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks and domains show MDCure consistently improves performance over pre-trained baselines and base models by up to 75.1%. Our code, datasets, and models are available at https://github.com/yale-nlp/MDCure.

Paper Structure

This paper contains 43 sections, 18 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MDCure
  4. Generation Phase
  5. Filtering Phase
  6. Experimental Setup
  7. Results
  8. Main Results
  9. Influence of Data Scale
  10. Generalization to Few-Shot & Extended-Context Training
  11. Efficacy with Open-Source Generation
  12. Integration with Policy Optimization
  13. Impact on Advanced Model Capabilities
  14. Conclusion
  15. Limitations
  16. ...and 28 more sections

Figures (18)

  • Figure 1: The MDCure pipeline generates diverse multi-document instructions, filters them using fine-grained scoring from MDCureRM, and tunes a base LLM to enhance its multi-document capabilities.
  • Figure 2: Fine-grained scoring criteria utilized by MDCureRM for RM-as-a-Judge evaluation of candidate instruction quality in the Filtering phase of MDCure.
  • Figure 3: Illustrative example of input format.
  • Figure 4: General Prompt Templates A-D
  • Figure 5: General Prompt Templates E-I
  • ...and 13 more figures