Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor

TL;DR

The paper argues that rigor in AI research has been narrowly equated with methodological thoroughness, which can lead to unsubstantiated claims and insufficient attention to societal and ethical implications. It proposes a broader, responsible AI-informed conception of rigor built around six facets—epistemic, normative, conceptual, methodological, reporting, and interpretative—to guide scrutiny by researchers, policymakers, journalists, and other stakeholders. For each facet, the authors delineate descriptive concerns and evaluative criteria, and outline mechanisms to promote rigor, such as explicit grounding, normative disclosures, conceptual clarity, construct validity, preregistration, and transparent documentation of artifacts and impacts. By framing rigor as a multi-faceted, dialogic scaffold rather than a rigid checklist, the work aims to enhance scientific integrity, accountability, and real-world relevance of AI research and practice.

Abstract

In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about the capabilities of AI systems. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders.

Figures (1)

• Figure 1: Simplified overview of the objects of concern for each facet of rigor and of common dependencies among them. For instance, the research findings typically determine what inferences and claims can be made, while normative considerations may influence the choices of theoretical constructs, of what methods to use, of what research findings to report, or of what inferences and claims to make. Dependencies are illustrated through both arrows as well as nested boxes.