Generative AI for Research Data Processing: Lessons Learnt From Three Use Cases

TL;DR

The paper investigates when and how generative AI, exemplified by Claude 3 Opus, can support research data processing across diverse tasks (information extraction, natural language understanding, and text classification). It introduces a disciplined pipeline with token-based chunking, deterministic prompts (temperature $0$), and JSON outputs to maximize accuracy and reproducibility, and it evaluates results against ground truth where available. Across three use cases, the approach yields near-perfect results for seedlists on a small sample, mixed but useful accuracy for HTA data extraction, and interrater-comparable performance for Kickstarter NAICS coding despite inherent task ambiguity. The work offers concrete guidance on selecting appropriate tasks for generative AI, highlights the importance of prompt design and data quality, and outlines a path toward more rigorous quantitative assessment and open, reproducible studies in the future.

Abstract

There has been enormous interest in generative AI since ChatGPT was launched in 2022. However, there are concerns about the accuracy and consistency of the outputs of generative AI. We have carried out an exploratory study on the application of this new technology in research data processing. We identified tasks for which rule-based or traditional machine learning approaches were difficult to apply, and then performed these tasks using generative AI. We demonstrate the feasibility of using the generative AI model Claude 3 Opus in three research projects involving complex data processing tasks: 1) Information extraction: We extract plant species names from historical seedlists (catalogues of seeds) published by botanical gardens. 2) Natural language understanding: We extract certain data points (name of drug, name of health indication, relative effectiveness, cost-effectiveness, etc.) from documents published by Health Technology Assessment organisations in the EU. 3) Text classification: We assign industry codes to projects on the crowdfunding website Kickstarter. We share the lessons we learnt from these use cases: How to determine if generative AI is an appropriate tool for a given data processing task, and if so, how to maximise the accuracy and consistency of the results obtained.

Figures (4)

• Figure 1: Accuracy and consistency of outputs: The red dot in the centre is the true value or the ground truth, and the three grey dots indicate outputs produced by different generative AI runs with the same input data, model, and parameters. For generative AI to be a reliable data processing method, we want our results to look like Fig. \ref{['fig:accurate_and_consistent']}, in which the outputs are both accurate and consistent.
• Figure 2: Generative AI pipeline
• Figure 3: Pages from seedlists in PDF format, from different Botanical Gardens, for the year 2020. These illustrate the diversity of seedlist formats. (\ref{['fig:sub12']}) Botanischer Garten der Universität Wien, Vienna, Austria (\ref{['fig:sub2']}) Hortus Botanicus, Academiae Scientiarum Bulgariae, Sofia, Bulgaria (\ref{['fig:sub9']}) Botanischer Garten und Arboretum, Linz, Austria
• Figure 4: A page from a scanned seedlist from the Botanical Garden of the University of Göttingen, 1970: (\ref{['scan']}) Original scanned text (\ref{['ocr']}) Text obtained by performing OCR on the scanned text. The OCR errors are highlighted in yellow.