The Human Factor in Data Cleaning: Exploring Preferences and Biases

TL;DR

Data cleaning decisions are shaped by human cognitive biases, which can distort data distributions before modeling. The study uses a controlled survey with census-inspired scenarios and $N=51$ participants to quantify bias mechanisms across error detection, repair/imputation, and entity matching, employing exact binomial tests against baselines $ au_0=0.5$ and $ au_0=0.10$. Key findings show robust manifestations of representativeness, framing, anchoring, and omission biases across tasks, while automation bias remains limited under a conservative tolerance. These results suggest that bias in data cleaning is structural and persists across expertise and workflow, motivating bias-aware human-in-the-loop designs that clearly separate representation from semantics and offer nonprescriptive, reversible recommendations to improve data quality and transparency.

Abstract

Data cleaning is often framed as a technical preprocessing step, yet in practice it relies heavily on human judgment. We report results from a controlled survey study in which participants performed error detection, data repair and imputation, and entity matching tasks on census-inspired scenarios with known semantic validity. We find systematic evidence for several cognitive bias mechanisms in data cleaning. Framing effects arise when surface-level formatting differences (e.g., capitalization or numeric presentation) increase false-positive error flags despite unchanged semantics. Anchoring and adjustment bias appears when expert cues shift participant decisions beyond parity, consistent with salience and availability effects. We also observe the representativeness heuristic: atypical but valid attribute combinations are frequently flagged as erroneous, and in entity matching tasks, surface similarity produces a substantial false-positive rate with high confidence. In data repair, participants show a robust preference for leaving values missing rather than imputing plausible values, consistent with omission bias. In contrast, automation-aligned switching under strong contradiction does not exceed a conservative rare-error tolerance threshold at the population level, indicating that deference to automated recommendations is limited in this setting. Across scenarios, bias patterns persist among technically experienced participants and across diverse workflow practices, suggesting that bias in data cleaning reflects general cognitive tendencies rather than lack of expertise. These findings motivate human-in-the-loop cleaning systems that clearly separate representation from semantics, present expert or algorithmic recommendations non-prescriptively, and support reflective evaluation of atypical but valid cases.

Figures (1)

• Figure 1: Radar summaries of baseline exceedance scores across bias--scenario measures. Each axis reports $E=100\cdot\max\!\left(0,(\hat{\tau}-\tau_0)/(1-\tau_0)\right)$, where $\hat{\tau}$ is the observed subgroup rate and $\tau_0 \in \{0.5,0.10\}$ is the scenario-specific tolerable baseline. Larger values indicate larger behavioral deviation above the tolerable threshold (not stronger statistical significance).