Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACL-rlg: A Dataset for Reading List Generation

Julien Aubert-Béduchaud, Florian Boudin, Béatrice Daille, Richard Dufour

TL;DR

The paper tackles the problem of efficiently familiarizing researchers with a field by introducing ACL-rlg, the largest expert-curated reading list dataset derived from ACL tutorials. It formalizes reading-list generation as an ordered retrieval task, defines evaluation benchmarks, and provides baselines using search engines and retrieval models as well as large language models. Findings show that while GPT-4o offers the best performance among tested systems, overall results are still limited and raise concerns about data contamination and hallucinations, underscoring the need for retrieval-augmented approaches and ordering-aware evaluation. The work contributes a valuable benchmark, releases code and data under MIT, and points toward future improvements in reading-list generation and evaluation metrics.

Abstract

Familiarizing oneself with a new scientific field and its existing literature can be daunting due to the large amount of available articles. Curated lists of academic references, or reading lists, compiled by experts, offer a structured way to gain a comprehensive overview of a domain or a specific scientific challenge. In this work, we introduce ACL-rlg, the largest open expert-annotated reading list dataset. We also provide multiple baselines for evaluating reading list generation and formally define it as a retrieval task. Our qualitative study highlights the fact that traditional scholarly search engines and indexing methods perform poorly on this task, and GPT-4o, despite showing better results, exhibits signs of potential data contamination.

ACL-rlg: A Dataset for Reading List Generation

TL;DR

The paper tackles the problem of efficiently familiarizing researchers with a field by introducing ACL-rlg, the largest expert-curated reading list dataset derived from ACL tutorials. It formalizes reading-list generation as an ordered retrieval task, defines evaluation benchmarks, and provides baselines using search engines and retrieval models as well as large language models. Findings show that while GPT-4o offers the best performance among tested systems, overall results are still limited and raise concerns about data contamination and hallucinations, underscoring the need for retrieval-augmented approaches and ordering-aware evaluation. The work contributes a valuable benchmark, releases code and data under MIT, and points toward future improvements in reading-list generation and evaluation metrics.

Abstract

Familiarizing oneself with a new scientific field and its existing literature can be daunting due to the large amount of available articles. Curated lists of academic references, or reading lists, compiled by experts, offer a structured way to gain a comprehensive overview of a domain or a specific scientific challenge. In this work, we introduce ACL-rlg, the largest open expert-annotated reading list dataset. We also provide multiple baselines for evaluating reading list generation and formally define it as a retrieval task. Our qualitative study highlights the fact that traditional scholarly search engines and indexing methods perform poorly on this task, and GPT-4o, despite showing better results, exhibits signs of potential data contamination.

Paper Structure

This paper contains 24 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Reading List Generation
  3. Dataset Description
  4. Reading Lists Extraction
  5. Query Annotation
  6. Experiments
  7. Results of Search Engines and LLMs
  8. Comparison of Retrieval Models on ACL Anthology Collection
  9. Analysis of LLMs Predictions
  10. Conclusion
  11. Limitations
  12. Ethical considerations
  13. Annotation Guidelines for creating queries
  14. Objective
  15. Dataset Source
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Distribution of tutorials by their number of reading list references.
  • Figure 2: Comparison of overall article popularity trends within the dataset reading lists (TOP) and the corresponding predicted outcomes (BOTTOM).
  • Figure 3: Recall@20 scores for each year of tutorials from which the reading lists were extracted.
  • Figure 4: % of references in Semantic Scholar.
  • Figure 5: Distribution of the number of referenced articles per year.