Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating measures of replicability into scholarly search: Challenges and opportunities

Chuhao Wu, Tatiana Chakravorti, John Carroll, Sarah Rajtmajer

TL;DR

This article investigates integrating signals of replicability into scholarly search, combining qualitative interviews with 17 social and behavioral science researchers and a demonstration of an AI-driven replicability estimator. It analyzes researchers’ literature-search practices, the meaning of replicability, and how automated scores might influence search, review, and design decisions. The authors identify confusion between replicability and generalizability, highlight the importance of transparent, context-aware explanations, and discuss ethical implications for AI-enabled confidence assessments. The work offers design-relevant insights for building domain-aware, credibility-incorporating search tools and outlines a research agenda to address explainability, uncertainty, and policy considerations in deploying replicability signals in scholarly workflows.

Abstract

Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability.

Integrating measures of replicability into scholarly search: Challenges and opportunities

TL;DR

This article investigates integrating signals of replicability into scholarly search, combining qualitative interviews with 17 social and behavioral science researchers and a demonstration of an AI-driven replicability estimator. It analyzes researchers’ literature-search practices, the meaning of replicability, and how automated scores might influence search, review, and design decisions. The authors identify confusion between replicability and generalizability, highlight the importance of transparent, context-aware explanations, and discuss ethical implications for AI-enabled confidence assessments. The work offers design-relevant insights for building domain-aware, credibility-incorporating search tools and outlines a research agenda to address explainability, uncertainty, and policy considerations in deploying replicability signals in scholarly workflows.

Abstract

Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability.
Paper Structure (29 sections, 3 figures, 1 table)

This paper contains 29 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Literature search and review
  4. The replication crisis
  5. Assessment of reproducibility and replicability
  6. AI prediction market for research confidence assessment
  7. Ethical considerations
  8. Methods
  9. Recruitment and Data Collection
  10. Data analysis
  11. Findings
  12. Literature search: Strategies and challenges
  13. Metrics for filtering search results
  14. Paper evaluation and information seeking
  15. Challenges to literature search
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Web interface for replicability estimation prototype tool. A: Home page; B: Interface for uploading PDFs for scoring; C: Interface for paper evaluation, with the ability for users to adjust model hyperparameters using Advanced Settings.
  • Figure 2: Example display of paper evaluation results. D: Replicability score. E: Explanations, presented as details about agents' behaviors/decisions; F: Publication's features in context; G: Visualization of principal components from extracted features.
  • Figure 3: Illustration of the study protocol.