Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicity

Prakhar Ganesh, Hsiang Hsu, Golnoosh Farnadi

TL;DR

This work studies predictive multiplicity through a neighbouring datasets lens, focusing on how single-data-point changes in data processing affect downstream multiplicity under a fixed Rashomon parameter $\epsilon$. The authors derive a theoretical result showing that higher inter-class overlap, as measured by the overlapping coefficient $OVL$, yields a smaller Rashomon set and thus lower multiplicity in neighbouring datasets, and they extend these insights to active learning and data imputation. They also introduce multiplicity-aware data acquisition and imputation methods (e.g., MultLow and MultHigh) that control multiplicity while maintaining accuracy, supported by empirical studies across multiple datasets. The proposed framework provides a practical, data-processing-centered approach to managing multiplicity and suggests future connections to differential privacy and robustness to distribution shifts and adversarial data.

Abstract

Multiplicity -- the existence of distinct models with comparable performance -- has received growing attention in recent years. While prior work has largely emphasized modelling choices, the critical role of data in shaping multiplicity has been comparatively overlooked. In this work, we introduce a neighbouring datasets framework to examine the most granular case: the impact of a single-data-point difference on multiplicity. Our analysis yields a seemingly counterintuitive finding: neighbouring datasets with greater inter-class distribution overlap exhibit lower multiplicity. This reversal of conventional expectations arises from a shared Rashomon parameter, and we substantiate it with rigorous proofs. Building on this foundation, we extend our framework to two practical domains: active learning and data imputation. For each, we establish natural extensions of the neighbouring datasets perspective, conduct the first systematic study of multiplicity in existing algorithms, and finally, propose novel multiplicity-aware methods, namely, multiplicity-aware data acquisition strategies for active learning and multiplicity-aware data imputation techniques.

Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicity

TL;DR

This work studies predictive multiplicity through a neighbouring datasets lens, focusing on how single-data-point changes in data processing affect downstream multiplicity under a fixed Rashomon parameter . The authors derive a theoretical result showing that higher inter-class overlap, as measured by the overlapping coefficient , yields a smaller Rashomon set and thus lower multiplicity in neighbouring datasets, and they extend these insights to active learning and data imputation. They also introduce multiplicity-aware data acquisition and imputation methods (e.g., MultLow and MultHigh) that control multiplicity while maintaining accuracy, supported by empirical studies across multiple datasets. The proposed framework provides a practical, data-processing-centered approach to managing multiplicity and suggests future connections to differential privacy and robustness to distribution shifts and adversarial data.

Abstract

Multiplicity -- the existence of distinct models with comparable performance -- has received growing attention in recent years. While prior work has largely emphasized modelling choices, the critical role of data in shaping multiplicity has been comparatively overlooked. In this work, we introduce a neighbouring datasets framework to examine the most granular case: the impact of a single-data-point difference on multiplicity. Our analysis yields a seemingly counterintuitive finding: neighbouring datasets with greater inter-class distribution overlap exhibit lower multiplicity. This reversal of conventional expectations arises from a shared Rashomon parameter, and we substantiate it with rigorous proofs. Building on this foundation, we extend our framework to two practical domains: active learning and data imputation. For each, we establish natural extensions of the neighbouring datasets perspective, conduct the first systematic study of multiplicity in existing algorithms, and finally, propose novel multiplicity-aware methods, namely, multiplicity-aware data acquisition strategies for active learning and multiplicity-aware data imputation techniques.
Paper Structure (39 sections, 1 theorem, 26 equations, 14 figures, 4 algorithms)

This paper contains 39 sections, 1 theorem, 26 equations, 14 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neighbouring Datasets
  4. Preliminaries: Rashomon Set and Multiplicity
  5. $k$-Neighbouring Datasets
  6. Higher Overlap leads to a Smaller Rashomon Set
  7. Theoretical Insights for Binary Classification
  8. Proof Sketch.
  9. Interpreting the Assumptions.
  10. Extending to $k$-Neighbouring Datasets
  11. Multiplicity and Active Learning
  12. Neighbouring Datasets in Active Learning
  13. Experiment Setup and Algorithms
  14. Controlling Multiplicity During Active Learning
  15. Multiplicity and Data Imputation
  16. ...and 24 more sections

Key Result

Theorem 4.1

Given two $1$-neighbouring binary classification datasets $D^1_{train}, D^2_{train}$ which, without loss of generality, differ only at the index $0$, i.e., $(x^1_0, y^1_0) \neq (x^2_0, y^2_0)$ and $(x^1_j, y^1_j) = (x^2_j, y^2_j) \; \forall j \neq 0$, and adhere to the following assumptions: then we can say that the overlapping coefficient between the two classes will be higher for this dataset,

Figures (14)

  • Figure 1: Our neighbouring datasets framework alongside model and dataset multiplicity frameworks.
  • Figure 2: Examples of neighbouring datasets in several data preparation and processing pipelines.
  • Figure 3: Spearman's rank correlation coefficients between the overlap and resulting multiplicity.
  • Figure 4: (a, b, c) Accuracy and ambiguity across various strategies for one step of data acquisition. We see clear trends of our MultLow (and MultHigh) approach(es) getting the lowest (and highest) multiplicity consistently, while maintaining similar accuracies. (d, e) Accuracy and ambiguity across multiple steps of data acquisition. Similar trends persist across multiple steps of active learning.
  • Figure 5: Correlation between the overlapping coefficient and resulting multiplicity.
  • ...and 9 more figures

Theorems & Definitions (6)

  • Definition 3.1: Rashomon Set
  • Definition 3.2: Ambiguity
  • Definition 3.3: $k$-Neighbouring Datasets
  • Definition 4.1: Overlapping Coefficient inman1989overlapping
  • Theorem 4.1
  • Conjecture 4.1