Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks

John Francis, Saba Esnaashari, Anton Poletaev, Sukankana Chakraborty, Youmna Hashem, Jonathan Bright

TL;DR

This paper introduces MIMDE, a two-step task for extracting an optimal set of insights from a document corpus and mapping them to source documents. It provides two complementary datasets (human-generated and synthetic) and a standardized evaluation framework to benchmark 20 state-of-the-art LLMs on MIMDE tasks, enabling direct comparison between human and synthetic data. Results show a strong, positive correlation ($r=0.71$) between models' ability to extract insights on both datasets, but synthetic data poorly models document-level mapping, highlighting both the potential and limits of synthetic data for evaluation. The work offers practical guidance for using synthetic data in evaluating text-analysis systems and outlines avenues for improving synthetic data generation and evaluation methodologies.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in text analysis tasks, yet their evaluation on complex, real-world applications remains challenging. We define a set of tasks, Multi-Insight Multi-Document Extraction (MIMDE) tasks, which involves extracting an optimal set of insights from a document corpus and mapping these insights back to their source documents. This task is fundamental to many practical applications, from analyzing survey responses to processing medical records, where identifying and tracing key insights across documents is crucial. We develop an evaluation framework for MIMDE and introduce a novel set of complementary human and synthetic datasets to examine the potential of synthetic data for LLM evaluation. After establishing optimal metrics for comparing extracted insights, we benchmark 20 state-of-the-art LLMs on both datasets. Our analysis reveals a strong correlation (0.71) between the ability of LLMs to extracts insights on our two datasets but synthetic data fails to capture the complexity of document-level analysis. These findings offer crucial guidance for the use of synthetic data in evaluating text analysis systems, highlighting both its potential and limitations.

MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks

TL;DR

This paper introduces MIMDE, a two-step task for extracting an optimal set of insights from a document corpus and mapping them to source documents. It provides two complementary datasets (human-generated and synthetic) and a standardized evaluation framework to benchmark 20 state-of-the-art LLMs on MIMDE tasks, enabling direct comparison between human and synthetic data. Results show a strong, positive correlation () between models' ability to extract insights on both datasets, but synthetic data poorly models document-level mapping, highlighting both the potential and limits of synthetic data for evaluation. The work offers practical guidance for using synthetic data in evaluating text-analysis systems and outlines avenues for improving synthetic data generation and evaluation methodologies.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in text analysis tasks, yet their evaluation on complex, real-world applications remains challenging. We define a set of tasks, Multi-Insight Multi-Document Extraction (MIMDE) tasks, which involves extracting an optimal set of insights from a document corpus and mapping these insights back to their source documents. This task is fundamental to many practical applications, from analyzing survey responses to processing medical records, where identifying and tracing key insights across documents is crucial. We develop an evaluation framework for MIMDE and introduce a novel set of complementary human and synthetic datasets to examine the potential of synthetic data for LLM evaluation. After establishing optimal metrics for comparing extracted insights, we benchmark 20 state-of-the-art LLMs on both datasets. Our analysis reveals a strong correlation (0.71) between the ability of LLMs to extracts insights on our two datasets but synthetic data fails to capture the complexity of document-level analysis. These findings offer crucial guidance for the use of synthetic data in evaluating text analysis systems, highlighting both its potential and limitations.

Paper Structure

This paper contains 20 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Information Extraction Tasks
  4. LLM Performance and Reliability
  5. Synthetic Data for Model Evaluation
  6. Methodology
  7. MIMDE Defintion
  8. Data Collection and Generation
  9. Human Data
  10. Synthetic Data
  11. Evaluation Framework
  12. Insight Similarity
  13. Metrics for Evaluating MIMDE
  14. Benchmark Implementation Approach
  15. Synthetic vs. Human Performance
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Evaluating Potential MIMDE Metrics (Precision) Measures how often true matches to human mapped insights are correctly identified. (Recall) Measures how well the model identifies true positives from all the true positives in the dataset. (F1) A weighted mean between Precision and Recall.
  • Figure 2: Relationship between Synthetic and Human data performance (Recall) Measures how often true insights have been correctly identified.