Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perspectives from India: Opportunities and Challenges for AI Replication Prediction to Improve Confidence in Published Research

Tatiana Chakravorti, Chuhao Wu, Sai Koneru, Sarah Rajtmajer

TL;DR

The paper investigates how Indian researchers perceive AI-based replication prediction to improve confidence in published findings, emphasizing the social, cultural, and institutional context that shapes adoption. Using 19 semi-structured interviews and a demonstrator replication-prediction prototype built on an artificial prediction-market framework, the authors identify a preference for hybrid human-AI systems, explainability, and domain-specific tool design. Key contributions include contextualized design implications for AI-driven research assessment, a qualitative account of openness practices in India, and policy-relevant recommendations to align incentives with reproducibility and transparency. The work highlights that effective deployment of such technologies requires governance, accessible artifacts, and credible interfaces that integrate smoothly with human peer review to enhance trust and uptake in research practice.

Abstract

Over the past decade, a crisis of confidence in scientific literature has gained attention, particularly in the West. In response, we have seen changes in policy and practice amongst individual researchers and institutions. Greater attention is given to the transparency of workflows and the appropriate use of statistical methods. Advances in scholarly big data and machine learning have led to the development of AI-driven tools for the evaluation of published findings. In this study, we conduct 19 semi-structured interviews with Indian researchers to understand their perspectives on challenges and opportunities for AI technologies to improve confidence in published research. Our findings highlight the importance of social and cultural context for the design and deployment of AI tools for research assessment. Our work suggests that such technologies must work alongside rather than replace human research assessment mechanisms. They must be explainable and situated within well-functioning human-centered peer review processes.

Perspectives from India: Opportunities and Challenges for AI Replication Prediction to Improve Confidence in Published Research

TL;DR

The paper investigates how Indian researchers perceive AI-based replication prediction to improve confidence in published findings, emphasizing the social, cultural, and institutional context that shapes adoption. Using 19 semi-structured interviews and a demonstrator replication-prediction prototype built on an artificial prediction-market framework, the authors identify a preference for hybrid human-AI systems, explainability, and domain-specific tool design. Key contributions include contextualized design implications for AI-driven research assessment, a qualitative account of openness practices in India, and policy-relevant recommendations to align incentives with reproducibility and transparency. The work highlights that effective deployment of such technologies requires governance, accessible artifacts, and credible interfaces that integrate smoothly with human peer review to enhance trust and uptake in research practice.

Abstract

Over the past decade, a crisis of confidence in scientific literature has gained attention, particularly in the West. In response, we have seen changes in policy and practice amongst individual researchers and institutions. Greater attention is given to the transparency of workflows and the appropriate use of statistical methods. Advances in scholarly big data and machine learning have led to the development of AI-driven tools for the evaluation of published findings. In this study, we conduct 19 semi-structured interviews with Indian researchers to understand their perspectives on challenges and opportunities for AI technologies to improve confidence in published research. Our findings highlight the importance of social and cultural context for the design and deployment of AI tools for research assessment. Our work suggests that such technologies must work alongside rather than replace human research assessment mechanisms. They must be explainable and situated within well-functioning human-centered peer review processes.
Paper Structure (46 sections, 1 figure, 1 table)

This paper contains 46 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. The replication crisis
  4. Transparency and open science
  5. AI for research assessment
  6. Methodology
  7. Participant recruitment
  8. Interview protocol
  9. An exemplar AI for research assessment
  10. Data analysis
  11. Ethics Statement
  12. Researcher Positionality
  13. Findings
  14. State of reproducibility and replicability in India
  15. Participant awareness.
  16. ...and 31 more sections

Figures (1)

  • Figure 1: Screen capture from a prototype research assessment tool. Separate tabs allow the user to explore the AI and its functionality, including extracted features from the paper of interest and a subset of similar papers from the training dataset.