Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMAQA: A Metadata-based QA Dataset for RAG Systems

Davide Bruni, Marco Avvenuti, Nicola Tonellotto, Maurizio Tesconi

TL;DR

AMAQA tackles the problem of evaluating QA in retrieval-augmented generation systems when textual data are enriched with structured metadata. It introduces a large-scale, open-access dataset built from public Telegram channels, combining text with metadata such as timestamps, topics, emotion, and toxicity, and pairs this with a high-quality QA subset. Through extensive experiments, the work shows that explicit metadata filtering dramatically improves retrieval accuracy ($0.12\to$ $0.61$) and that retrieval reranking, iterative context expansion, and controlled noise can push accuracy to $0.75$ with a NEM of $0.54$, establishing a new state of the art for metadata-driven QA. The dataset and methodology offer a practical foundation for metadata-aware QA in real-world domains and pave the way for further enhancements via embedding/LMM fine-tuning and domain expansion.

Abstract

Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, hindering evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps, topics, emotional tones, and toxicity indicators, which enable precise and contextualized queries by filtering documents based on specific criteria. It also includes 450 high-quality QA pairs, making it a valuable resource for advancing research on metadata-driven QA and RAG systems. To the best of our knowledge, AMAQA is the first single-hop QA benchmark to incorporate metadata and labels such as topics covered in the messages. We conduct extensive tests on the benchmark, establishing a new standard for future research. We show that leveraging metadata boosts accuracy from 0.12 to 0.61, highlighting the value of structured context. Building on this, we explore several strategies to refine the LLM input by iterating over provided context and enriching it with noisy documents, achieving a further 3-point gain over the best baseline and a 14-point improvement over simple metadata filtering. The dataset is available at https://anonymous.4open.science/r/AMAQA-5D0D/

AMAQA: A Metadata-based QA Dataset for RAG Systems

TL;DR

AMAQA tackles the problem of evaluating QA in retrieval-augmented generation systems when textual data are enriched with structured metadata. It introduces a large-scale, open-access dataset built from public Telegram channels, combining text with metadata such as timestamps, topics, emotion, and toxicity, and pairs this with a high-quality QA subset. Through extensive experiments, the work shows that explicit metadata filtering dramatically improves retrieval accuracy ( ) and that retrieval reranking, iterative context expansion, and controlled noise can push accuracy to with a NEM of , establishing a new state of the art for metadata-driven QA. The dataset and methodology offer a practical foundation for metadata-aware QA in real-world domains and pave the way for further enhancements via embedding/LMM fine-tuning and domain expansion.

Abstract

Retrieval-augmented generation (RAG) systems are widely used in question-answering (QA) tasks, but current benchmarks lack metadata integration, hindering evaluation in scenarios requiring both textual data and external information. To address this, we present AMAQA, a new open-access QA dataset designed to evaluate tasks combining text and metadata. The integration of metadata is especially important in fields that require rapid analysis of large volumes of data, such as cybersecurity and intelligence, where timely access to relevant information is critical. AMAQA includes about 1.1 million English messages collected from 26 public Telegram groups, enriched with metadata such as timestamps, topics, emotional tones, and toxicity indicators, which enable precise and contextualized queries by filtering documents based on specific criteria. It also includes 450 high-quality QA pairs, making it a valuable resource for advancing research on metadata-driven QA and RAG systems. To the best of our knowledge, AMAQA is the first single-hop QA benchmark to incorporate metadata and labels such as topics covered in the messages. We conduct extensive tests on the benchmark, establishing a new standard for future research. We show that leveraging metadata boosts accuracy from 0.12 to 0.61, highlighting the value of structured context. Building on this, we explore several strategies to refine the LLM input by iterating over provided context and enriching it with noisy documents, achieving a further 3-point gain over the best baseline and a 14-point improvement over simple metadata filtering. The dataset is available at https://anonymous.4open.science/r/AMAQA-5D0D/
Paper Structure (20 sections, 8 figures, 3 tables)

This paper contains 20 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The AMAQA dataset
  4. Data collection
  5. Data labeling
  6. Dataset statistics
  7. QA creation
  8. Ethical consideration
  9. RAG systems evaluation methodology
  10. RAG systems: baselines
  11. RAG systems: novel approaches
  12. Evaluation metrics
  13. Experimental setup
  14. Experimental Evaluation of RAG Systems with AMAQA
  15. Impact of Metadata on RAG Performance
  16. ...and 5 more sections

Figures (8)

  • Figure 1: High-level conceptual schema of the systems analyzed in this work
  • Figure 2: Visualization of message and topic distributions, providing insights into temporal trends and the division of messages by topic.
  • Figure 3: Distribution of emotions across messages, illustrating the prevalence of different emotional tones.
  • Figure 4: RAG systems capable of handling metadata schema. The optional component is part of the $\text{Re}^2\text{G}$ architecture and all its derived approaches.
  • Figure 5: Generator component schema in the Iter-$\text{Re}^2\text{G}$ architecture: after the first response, the system checks for "I don't know." If found, it uses the next top_n documents for a new generation; otherwise, it accepts the response. This repeats until a valid response is given or no documents remain.
  • ...and 3 more figures