Mitigating Omitted Variable Bias in Empirical Software Engineering

TL;DR

The paper addresses omitted variable bias in observational empirical software engineering and argues that causal structural models (DAGs) combined with adjustment sets and sensitivity analyses provide robust strategies to identify, quantify, and mitigate bias. It demonstrates two illustrative cases—programming language effects on code quality and team size effects on effort—to show how unmeasured confounders can distort causal inferences and how DAG-driven adjustments or tipping-point/E-value analyses can restore or bound validity. A practical workflow is presented: define variables and causal structure, determine adjustment sets, and apply sensitivity analyses or simulations when unmeasured confounding cannot be fully resolved. The work contributes actionable guidelines and a replication package to help researchers design more valid observational studies in software engineering and beyond.

Abstract

Omitted variable bias occurs when a statistical model leaves out variables that are relevant determinants of the effects under study. This results in the model attributing the missing variables' effect to some of the included variables -- hence over- or under-estimating the latter's true effect. Omitted variable bias presents a significant threat to the validity of empirical research, particularly in non-experimental studies such as those prevalent in empirical software engineering. This paper illustrates the impact of omitted variable bias on two illustrative examples in the software engineering domain, and uses them to present methods to investigate the possible presence of omitted variable bias, to estimate its impact, and to mitigate its drawbacks. The analysis techniques we present are based on causal structural models of the variables of interest, which provide a practical, intuitive summary of the key relations among variables. This paper demonstrates a sequence of analysis steps that inform the design and execution of any empirical study in software engineering. An important observation is that it pays off to invest effort investigating omitted variable bias before actually executing an empirical study, because this effort can lead to a more solid study design, and to a significant reduction in its threats to validity.

Figures (11)

• Figure 1: Three possible processes where variable $Y$ depends on variables $X$ and $Z$, and the corresponding DAGs capturing these structural relations.
• Figure 2: Two linear regression models that capture the dependence between $X$, $Y$, and $Z$.
• Figure 3: The effect of $\textnormal{Language}$ on code $\textnormal{Quality}$ is confounded by the programmer's $\textnormal{Skill}$.
• Figure 4: Different models to estimate the dependence between $\textnormal{Quality}$ and $\textnormal{Language}$.
• Figure 5: A graphical summary of tipping analyses of confounder effect of $\textnormal{Skill}$ on the $\textnormal{Language} \to \textnormal{Quality}$ effect. For different measured values of effect ($\beta_{\textnormal{Language} \to \textnormal{Quality}}$), the plot shows the values of the effect $\textnormal{Skill} \to \textnormal{Quality}$ ($\beta_{\textnormal{Skill} \to \textnormal{Quality}}$) and of the scd$\textnormal{Skill} \to \textnormal{Language}$ ($\textsf{smd}_{\textnormal{Skill} \to \textnormal{Language}}$) that would tip the main effect. The two blue cross marks correspond to the two scenarios discussed in \ref{['sec:tip-SMD']} and \ref{['sec:tip-effect']}.