[Citation needed] Data usage and citation practices in medical imaging conferences

TL;DR

This work tackles the lack of standardized dataset usage and citation practices in medical imaging by introducing an automated pipeline that leverages OpenAlex and full-text analysis, plus a PDF annotation tool for validation. It applies these tools to study 20 public datasets across MICCAI and MIDL conferences from 2013–2023, revealing that a small subset of datasets dominate usage and that citation practices are inconsistent, with many papers either citing without mentioning or mentioning without proper citation. The authors discuss limitations such as data availability, versioning, and reliance on regex-based detection, and advocate for explicit data-availability sections to improve trackability and reproducibility in medical imaging research.

Abstract

Medical imaging papers often focus on methodology, but the quality of the algorithms and the validity of the conclusions are highly dependent on the datasets used. As creating datasets requires a lot of effort, researchers often use publicly available datasets, there is however no adopted standard for citing the datasets used in scientific papers, leading to difficulty in tracking dataset usage. In this work, we present two open-source tools we created that could help with the detection of dataset usage, a pipeline \url{https://github.com/TheoSourget/Public_Medical_Datasets_References} using OpenAlex and full-text analysis, and a PDF annotation software \url{https://github.com/TheoSourget/pdf_annotator} used in our study to manually label the presence of datasets. We applied both tools on a study of the usage of 20 publicly available medical datasets in papers from MICCAI and MIDL. We compute the proportion and the evolution between 2013 and 2023 of 3 types of presence in a paper: cited, mentioned in the full text, cited and mentioned. Our findings demonstrate the concentration of the usage of a limited set of datasets. We also highlight different citing practices, making the automation of tracking difficult.

Figures (19)

• Figure 1: Pipeline to detect dataset presence and usage. Green CSV represents user input, blue CSV represents extracted data
• Figure 2: Cumulative counts per year of dataset citations (full line) and mentions (dashed line) for classification datasets (a) and segmentation datasets (b).
• Figure 3: Type of presence per dataset and venue. The number in [] indicates the total number of papers for this subset. The "Only Cited" group in blue represents papers that cite a dataset without having a mention detected and therefore may not use it. The "Only Mentioned" group in orange represent the bad citation practice as the usage would not be detected by tools tracking the citations. The "Cited and Mentioned" group in green represent the best practice.
• Figure 4: Cumulative counts per year of dataset citations (full line) and mentions (dashed line) for classification datasets (a) and segmentation datasets (b).
• Figure 5: Type of presence per dataset and venue. The number in [] indicates the total number of papers for this subset.