Why are there many equally good models? An Anatomy of the Rashomon Effect

TL;DR

The paper tackles why many equally good models exist (the Rashomon effect) and its implications for inference, interpretability, fairness, and decision-making. It develops a formal framework built around the Rashomon set $\mathcal{R}(\epsilon,\mathcal{F},\mathcal{D})$ and Rashomon ratio $\rho(\epsilon,\mathcal{F},\mathcal{D})$, and introduces metrics across size, prediction, explanation, and information-theoretic dimensions. It organizes the causes of multiplicity into statistical, structural, and procedural sources, linking structural multiplicity to partial identification and discussing how these sources interact and materialize in practice. The work highlights practical opportunities, such as uncertainty quantification and diverse model explanations, while outlining methodological directions (e.g., Rashomon-set enumeration tools) and stability concerns for high-stakes settings. Overall, the Rashomon effect is presented as a fundamental facet of learning from data, whose careful analysis can improve robustness, transparency, and principled decision-making.

Abstract

The Rashomon effect -- the existence of multiple, distinct models that achieve nearly equivalent predictive performance -- has emerged as a fundamental phenomenon in modern machine learning and statistics. In this paper, we explore the causes underlying the Rashomon effect, organizing them into three categories: statistical sources arising from finite samples and noise in the data-generating process; structural sources arising from non-convexity of optimization objectives and unobserved variables that create fundamental non-identifiability; and procedural sources arising from limitations of optimization algorithms and deliberate restrictions to suboptimal model classes. We synthesize insights from machine learning, statistics, and optimization literature to provide a unified framework for understanding why the multiplicity of good models arises. A key distinction emerges: statistical multiplicity diminishes with more data, structural multiplicity persists asymptotically and cannot be resolved without different data or additional assumptions, and procedural multiplicity reflects choices made by practitioners. Beyond characterizing causes, we discuss both the challenges and opportunities presented by the Rashomon effect, including implications for inference, interpretability, fairness, and decision-making under uncertainty.

Figures (3)

• Figure 1: Statistical sources of the Rashomon effect arise from finite samples and noise. Left: With noisy data, fundamentally different models (linear, quadratic, logarithmic) may fit equally well and be statistically indistinguishable. Right: Even when the true loss $L_{\text{pop}}(\theta)$ has a unique minimum $\theta^*$, finite samples create an indistinguishability region where many models achieve similar empirical performance. This region shrinks as sample size increases.
• Figure 2: Structural sources of the Rashomon effect arise from non-convex optimization landscapes with multiple local minima (left) and unobserved confounders creating non-identifiability (right). This multiplicity persists regardless of sample size.
• Figure 3: Procedural sources of the Rashomon effect arise from optimizer limitations such as initialization dependence (left) and deliberate restrictions to suboptimal model classes (right). These reflect practitioner choices that could in principle be altered.

Theorems & Definitions (6)

• Definition 1: Rashomon Set
• Definition 2: Rashomon Ratio
• Example 1: Linear Regression with Collinear Predictors
• Example 2: Decision Trees
• Example 3: Neural Networks
• Example 4: Cross-Model-Class Multiplicity