Data-NoMAD: A Tool for Boosting Confidence in the Integrity of Social Science Survey Data
Sanford C. Gordon, Cyrus Samii, Zhihao Su
TL;DR
The paper addresses the fragility of trust in social science data due to non-transparent data preparation and potential manipulation. It proposes Data-NoMAD, a cryptographic, ex-ante integrity framework that uses per-column $SHA-256$ digests to certify raw survey data at collection and to enable verifiable checks of archived datasets in replication archives. The key contributions include a two-mode system (Digest and Verify), a detailed architectural blueprint (UI, backend, and cloud storage), and a discussion of vulnerabilities and best practices to deter manipulation while preserving legitimate data handling. The approach aims to deter fraud, reduce reliance on costly ex-post audits, and enhance trust in replication efforts, with immediate applicability to survey data and potential extension to other data types in future work.
Abstract
To safeguard against data fabrication and enhance trust in quantitative social science, we present Data Non-Manipulation Authentication Digest (Data-NoMAD). Data-NoMAD is a tool that allows researchers to certify, and others to verify, that a dataset has not been inappropriately manipulated between the point of data collection and the point at which a replication archive is made publicly available. Data-NoMAD creates and stores a column hash digest of a raw dataset upon initial download from a survey platform (the current version works with Qualtrics and SurveyCTO), but before it is subject to appropriate manipulations such as anonymity-preserving redactions. Data-NoMAD can later be used to verify the integrity of a publicly archived dataset by identifying columns that have been deleted, added, or altered. Data-NoMAD complements existing efforts at ensuring research integrity and integrates seamlessly with extant replication practices.