Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updates
Ayoub Ajarra, Debabrota Basu
TL;DR
This work studies fairness auditing under adaptive model updates, introducing a PAC auditing framework powered by an Empirical Property Optimization (EPO) oracle. It shifts focus from reconstructing the original model to identifying a prospect class of property-preserving post-update models, and introduces the SP-dimension to quantify the complexity of admissible updates for Statistical Parity. The framework provides finite-class sample guarantees and, for infinite classes, demonstrates the need for SP-dimension-based analysis since infinite VC classes are not auditable under SP. Empirical results on COMPAS and Student datasets show accurate reconstruction of the prospect class and reliable SP estimates with favorable runtimes, illustrating practical applicability for ongoing fairness monitoring amidst model drift.
Abstract
As machine learning models become increasingly embedded in societal infrastructure, auditing them for bias is of growing importance. However, in real-world deployments, auditing is complicated by the fact that model owners may adaptively update their models in response to changing environments, such as financial markets. These updates can alter the underlying model class while preserving certain properties of interest, raising fundamental questions about what can be reliably audited under such shifts. In this work, we study group fairness auditing under arbitrary updates. We consider general shifts that modify the pre-audit model class while maintaining invariance of the audited property. Our goals are two-fold: (i) to characterize the information complexity of allowable updates, by identifying which strategic changes preserve the property under audit; and (ii) to efficiently estimate auditing properties, such as group fairness, using a minimal number of labeled samples. We propose a generic framework for PAC auditing based on an Empirical Property Optimization (EPO) oracle. For statistical parity, we establish distribution-free auditing bounds characterized by the SP dimension, a novel combinatorial measure that captures the complexity of admissible strategic updates. Finally, we demonstrate that our framework naturally extends to other auditing objectives, including prediction error and robust risk.