DIVERSE: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration

TL;DR

DIVERSE addresses predictive multiplicity by efficiently exploring the local Rashomon set around a fixed reference model without retraining or gradients. It couples Feature-wise Linear Modulation (FiLM) layers with a shared latent vector ${\bm{z}}$ to define a low-dimensional modulation space, and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to optimize for maximum disagreement while enforcing accuracy within a tolerance $\epsilon$. The approach yields diverse, high-performing model variants on MNIST, PneumoniaMNIST, and CIFAR-10, with competitive efficiency relative to retraining and improved diversity over dropout baselines, as validated on held-out test data and quantified by metrics such as Discrepancy, Ambiguity, VPR, and RC. These findings demonstrate a practical pathway for multiplicity analysis and robust, diverse model sets in deep networks, including initial evidence of applicability to Vision Transformers.

Abstract

We propose DIVERSE, a framework for systematically exploring the Rashomon set of deep neural networks, the collection of models that match a reference model's accuracy while differing in their predictive behavior. DIVERSE augments a pretrained model with Feature-wise Linear Modulation (FiLM) layers and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search a latent modulation space, generating diverse model variants without retraining or gradient access. Across MNIST, PneumoniaMNIST, and CIFAR-10, DIVERSE uncovers multiple high-performing yet functionally distinct models. Our experiments show that DIVERSE offers a competitive and efficient exploration of the Rashomon set, making it feasible to construct diverse sets that maintain robustness and performance while supporting well-balanced model multiplicity. While retraining remains the baseline to generate Rashomon sets, DIVERSE achieves comparable diversity at reduced computational cost.

Figures (9)

• Figure 1: FiLM placement (blue): (a) after dense layers, (b) after convolutional blocks, after batch normalization when present, otherwise directly after the convolution, and (c) on residual skip connections. A shared latent vector ${\bm{z}}$ provides the modulation parameters for all FiLM layers per model.
• Figure 2: Diversity metrics (discrepancy, ambiguity, Rashomon capacity, VPR) and Rashomon Ratio across Rashomon thresholds ($\epsilon$) for varying latent dimensions ($d$). Each group of points corresponds to one $\epsilon$; horizontal spacing within groups is only for readability and carries no semantic meaning. Markers show medians, error bars the interquartile range (IQR) across latent initializations, CMA-ES step sizes ($\sigma_0$), and $d$.
• Figure 3: Highest-disagreement samples for MNIST, CIFAR-10, and PneumoniaMNIST. Each panel displays the input image and the class-frequency distribution over the Rashomon set members; the true class is shown in green, and all other classes are represented in gray.
• Figure 4: Best hyperparameter configuration for DIVERSE compared against the pre-defined parameters of the dropout authors vs retraining.
• Figure 5: Ablation over the mixing weight $\lambda$. Across all datasets, diversity and Rashomon-set metrics change only marginally over $\lambda \in {0.0, 0.25, 0.5, 0.75, 1.0}$. Median curves and IQR bands are computed over different ${\bm{z}}_0$ initializations of the FiLM latent vector.