The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning

TL;DR

The paper investigates how the multiplicity of near-optimal models in Rashomon sets affects trustworthiness in high-stakes ML, showing that diversity can enable reactive robustness and stability but also increase information leakage. The authors develop theoretical results and empirical evidence on sparse decision trees and linear models, highlighting a robustness–privacy trade-off that depends on set diversity. They demonstrate that single sparse models are private yet fragile, while diverse Rashomon sets can sustain accuracy under targeted attacks but leak more training-data information when disclosed. The findings motivate governance at the Rashomon-set level, suggesting policies to balance transparency with privacy in practical deployments.

Abstract

Real-world machine learning (ML) pipelines rarely produce a single model; instead, they produce a Rashomon set of many near-optimal ones. We show that this multiplicity reshapes key aspects of trustworthiness. At the individual-model level, sparse interpretable models tend to preserve privacy but are fragile to adversarial attacks. In contrast, the diversity within a large Rashomon set enables reactive robustness: even when an attack breaks one model, others often remain accurate. Rashomon sets are also stable under small distribution shifts. However, this same diversity increases information leakage, as disclosing more near-optimal models provides an attacker with progressively richer views of the training data. Through theoretical analysis and empirical studies of sparse decision trees and linear models, we characterize this robustness-privacy trade-off and highlight the dual role of Rashomon sets as both a resource and a risk for trustworthy ML.

Figures (15)

• Figure 1: The robustness–privacy trade-off on the COMPAS dataset. As more diverse models from the Rashomon set are included, adversarial accuracy increases (greater reactive robustness), while reconstruction error decreases (greater information leakage).
• Figure 2: KL divergence between $p(x)$ and $q_{\Pi}(x)$ decreases as more trees are released into the ensemble.
• Figure 3: Adversarial score (accuracy) of trees in the Rashomon set vs. their distance to the optimal tree. Results are aggregated over five folds. The optimal trees (in red) are attacked. The most robust trees (in purple) are far from the optimal tree. Trees with the same distance to optimal trees are grouped, and mean and standard deviation of their adversarial score are shown as line plots with shaded uncertainty.
• Figure 4: Comparison of reconstruction error between different selection strategies. The random baseline randomly guesses the feature values for each data point.
• Figure 5: Reconstruction error vs. adversarial accuracy for ensembles constructed with different numbers of evenly sampled trees from the Rashomon set.

Theorems & Definitions (32)

• Theorem 1: Sparsity controls mutual information in a single tree
• Theorem 2: Inherent vulnerability of single models
• Theorem 3: Risk on adversarial dataset
• Corollary 1
• Corollary 2
• Theorem 4: Rashomon set is robust under small distribution shift
• Theorem 5: Two Rashomon sets constructed on neighboring datasets are indistinguishable
• Theorem 6: KL divergence bound for random ensembles from the Rashomon set
• Theorem \ref{th:rule_list_attack}: Inherent vulnerability of single models
• proof