Quantifying Model Uniqueness in Heterogeneous AI Ecosystems

TL;DR

A statistical framework for auditing model uniqueness based on In-Silico Quasi-Experimental Design (ISQED), which isolate intrinsic model identity and quantify uniqueness as the Peer-Inexpressible Residual (PIER), and establishes a principled, intervention-based science of auditing and governing heterogeneous model ecosystems.

Abstract

As AI systems evolve from isolated predictors into complex, heterogeneous ecosystems of foundation models and specialized adapters, distinguishing genuine behavioral novelty from functional redundancy becomes a critical governance challenge. Here, we introduce a statistical framework for auditing model uniqueness based on In-Silico Quasi-Experimental Design (ISQED). By enforcing matched interventions across models, we isolate intrinsic model identity and quantify uniqueness as the Peer-Inexpressible Residual (PIER), i.e. the component of a target's behavior strictly irreducible to any stochastic convex combination of its peers, with vanishing PIER characterizing when such a routing-based substitution becomes possible. We establish the theoretical foundations of ecosystem auditing through three key contributions. First, we prove a fundamental limitation of observational logs: uniqueness is mathematically non-identifiable without intervention control. Second, we derive a scaling law for active auditing, showing that our adaptive query protocol achieves minimax-optimal sample efficiency ($dσ^2γ^{-2}\log(Nd/δ)$). Third, we demonstrate that cooperative game-theoretic methods, such as Shapley values, fundamentally fail to detect redundancy. We implement this framework via the DISCO (Design-Integrated Synthetic Control) estimator and deploy it across diverse ecosystems, including computer vision models (ResNet/ConvNeXt/ViT), large language models (BERT/RoBERTa), and city-scale traffic forecasters. These results move trustworthy AI beyond explaining single models: they establish a principled, intervention-based science of auditing and governing heterogeneous model ecosystems.

Figures (5)

• Figure 1: Quantifying uniqueness capacity via in-silico quasi-experimental design.a, The auditing problem arises in heterogeneous ecosystems where a target model ($M_t$, red) must be evaluated against existing peers (blue) to detect functional redundancy. b, The in-silico quasi-experimental design (ISQED) protocol. By applying matched Type-B interventions (input perturbations, green) across all models, the framework isolates the Type-A effect (intrinsic model identity) from environmental variation. c, Geometrically, uniqueness is quantified as the peer-inexpressible residual (PIER, $R_t$), defined as the orthogonal projection residual of the target response onto the convex hull of peer responses ($\mathcal{C}$). d, While passive observation is inefficient, the in-silico setting permits active auditing. Our theory derives a fundamental scaling law (red curve), showing that uniqueness can be detected with minimax-optimal sample efficiency using adaptive queries.
• Figure 2: DISCO and PIER: auditing, attribution, and ecosystem effects. (A) Active auditing reduces query complexity compared to passive sampling, achieving a $1.34\times$ reduction at 5% error. (B) Attribution versus auditing in a toy ecosystem: Shapley assigns positive value to a redundant model, while PIER assigns near-zero uniqueness and correctly identifies a niche, non-substitutable model. (C) Ecosystem saturation: as the number of peer models $N$ grows beyond the intrinsic task dimension $d$ (here $d=10$), average uniqueness (PIER) collapses, revealing diminishing returns from adding peers.
• Figure 3: Auditing an SST-2 ecosystem with DISCO under matched token masking. PIER reveals dose-dependent uniqueness (A,B) that can differ from raw disagreement (C). Context fingerprints at $\theta=0.0$ and $0.5$ localize which linguistic regimes drive residuals (D). Routing checks validate the interpretation: oracle routing still fails for ALBERT at $\theta=0.0$, while DISCO-weight convex routing matches all targets at $\theta=0.5$ (E). Controlled targets separate redundancy, architectural divergence, and parametric divergence (F).
• Figure 4: Auditing vision ecosystems under controlled stressors. (A) Relative uniqueness across a matched adversarial dose $\epsilon$, reported as PIER$(\epsilon)$/PIER$(0)$, with baseline PIER$(0)$ shown at right. (B) Context audit comparing texture-natural versus shape-biased evaluation, plus the within-model amplification $\log_2(\mathrm{PIER}_{\mathrm{shape}}/\mathrm{PIER}_{\mathrm{nat}})$. (C) Context-activated uniqueness $\Delta$PIER decouples from both inductive bias (shape bias score) and a utility proxy (ImageNet top-1). (D) Heavy-tail diagnostics show that a small set of shared outliers dominates residual mass under shape-bias. (E) Convex-hull geometry: targets move outside the peer hull in the shape-bias context, and DISCO weights explain the closest convex surrogate.
• Figure 5: Auditing traffic models trained on multiple cities. (A) Relative uniqueness versus signed replacement impact when substituting a local city model with the DISCO convex router (inset shows the dense region; marker shape indicates whether GLOBAL beats the local model). (B) Convexity gap between a pairwise proxy and ecosystem uniqueness (left), with DISCO weights that explain representative high-gap cases (right). (C) Geometric signature linking uniqueness to convex-mix complexity; we report Spearman's rank correlation $\rho_s$. (D) Consolidation curve: system MAE increase as we prune city models under different pruning orders. Utility-only ranks cities by their standalone forecasting skill relative to a persistence baseline, ignoring substitution/replacement penalties. Oracle (penalty) ranks cities using access to the true replacement penalty (available only for evaluation, not in practice). (E) Tail-risk bridge: worst-case and tail excess degradation along the pruning path.

Theorems & Definitions (24)

• Proposition 1: Consistency under fixed designs
• proof
• Proposition 2: Asymptotic normality under fixed designs
• proof
• Proposition 3: Finite-sample design error bound
• proof
• Proposition 4: Monotonicity in the peer set
• proof
• Proposition 5: Conservatism of convex peer aggregation
• proof