Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RegCheck: A tool for automating comparisons between study registrations and papers

Jamie Cummins, Beth Clarke, Ian Hussey, Malte Elson

TL;DR

This paper addresses the burden of verifying study registrations against published reports, a key factor in research transparency and trustworthiness. It introduces RegCheck, a modular, LLM-assisted workflow that automates extraction of registration and publication content, supports user-defined comparison dimensions, and delivers auditable, shareable reports while preserving human oversight. The authors detail a six-stage pipeline—Ingestion, Extraction, Embedding, Definition, Analysis, and Reporting—along with design principles that emphasize practicality, non-prescriptiveness, and discipline-agnostic applicability, plus privacy considerations. They also discuss evaluation strategies, recognizing challenges in defining ground truth and proposing interrater agreement-based metrics, and position RegCheck as extensible infrastructure poised to enhance reproducibility across editorial, peer-review, and synthesis workflows, with public access through a web app and open-source code.

Abstract

Across the social and medical sciences, researchers recognize that specifying planned research activities (i.e., 'registration') prior to the commencement of research has benefits for both the transparency and rigour of science. Despite this, evidence suggests that study registrations frequently go unexamined, minimizing their effectiveness. In a way this is no surprise: manually checking registrations against papers is labour- and time-intensive, requiring careful reading across formats and expertise across domains. The advent of AI unlocks new possibilities in facilitating this activity. We present RegCheck, a modular LLM-assisted tool designed to help researchers, reviewers, and editors from across scientific disciplines compare study registrations with their corresponding papers. Importantly, RegCheck keeps human expertise and judgement in the loop by (i) ensuring that users are the ones who determine which features should be compared, and (ii) presenting the most relevant text associated with each feature to the user, facilitating (rather than replacing) human discrepancy judgements. RegCheck also generates shareable reports with unique RegCheck IDs, enabling them to be easily shared and verified by other users. RegCheck is designed to be adaptable across scientific domains, as well as registration and publication formats. In this paper we provide an overview of the motivation, workflow, and design principles of RegCheck, and we discuss its potential as an extensible infrastructure for reproducible science with an example use case.

RegCheck: A tool for automating comparisons between study registrations and papers

TL;DR

This paper addresses the burden of verifying study registrations against published reports, a key factor in research transparency and trustworthiness. It introduces RegCheck, a modular, LLM-assisted workflow that automates extraction of registration and publication content, supports user-defined comparison dimensions, and delivers auditable, shareable reports while preserving human oversight. The authors detail a six-stage pipeline—Ingestion, Extraction, Embedding, Definition, Analysis, and Reporting—along with design principles that emphasize practicality, non-prescriptiveness, and discipline-agnostic applicability, plus privacy considerations. They also discuss evaluation strategies, recognizing challenges in defining ground truth and proposing interrater agreement-based metrics, and position RegCheck as extensible infrastructure poised to enhance reproducibility across editorial, peer-review, and synthesis workflows, with public access through a web app and open-source code.

Abstract

Across the social and medical sciences, researchers recognize that specifying planned research activities (i.e., 'registration') prior to the commencement of research has benefits for both the transparency and rigour of science. Despite this, evidence suggests that study registrations frequently go unexamined, minimizing their effectiveness. In a way this is no surprise: manually checking registrations against papers is labour- and time-intensive, requiring careful reading across formats and expertise across domains. The advent of AI unlocks new possibilities in facilitating this activity. We present RegCheck, a modular LLM-assisted tool designed to help researchers, reviewers, and editors from across scientific disciplines compare study registrations with their corresponding papers. Importantly, RegCheck keeps human expertise and judgement in the loop by (i) ensuring that users are the ones who determine which features should be compared, and (ii) presenting the most relevant text associated with each feature to the user, facilitating (rather than replacing) human discrepancy judgements. RegCheck also generates shareable reports with unique RegCheck IDs, enabling them to be easily shared and verified by other users. RegCheck is designed to be adaptable across scientific domains, as well as registration and publication formats. In this paper we provide an overview of the motivation, workflow, and design principles of RegCheck, and we discuss its potential as an extensible infrastructure for reproducible science with an example use case.
Paper Structure (8 sections, 1 figure)

This paper contains 8 sections, 1 figure.

Table of Contents

  1. Introduction
  2. The RegCheck Workflow
  3. Design Principles and Philosophy
  4. Privacy and Confidentiality
  5. Is RegCheck Available To Use?
  6. When and How Should I Use RegCheck?
  7. How Accurate Is RegCheck?
  8. Future Agenda

Figures (1)

  • Figure 1: A screenshot of an example RegCheck report. Each row represents a comparison between the registration and paper along a specific, pre-specified dimension (noted in the 'Dimension' column). Under the 'Preregistration' and 'Paper' columns, users can toggle between displaying direct quotes from the respective document, or LLM-generated summaries of the content of these quotes. The color-coding of the row indicates whether a deviation between the materials was found (red), no deviation was found (blue), or that there was insufficient information in one or both sources to render a deviation judgement (yellow). The 'Deviation Information' column also provides commentary on the similarities and differences between the two sets of materials.