Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Active Fourier Auditor for Estimating Distributional Properties of ML Models

Ayoub Ajarra, Bishwamittra Ghosh, Debabrota Basu

TL;DR

The paper addresses auditing three distributional properties of black-box classifiers—robustness, individual fairness, and group fairness—by avoiding full model reconstruction and instead exploiting a Fourier expansion of the classifier. It introduces the Active Fourier Auditor (AFA), which adaptively queries inputs to recover significant Fourier coefficients via a Goldreich–Levin style procedure and then estimates the properties with PAC guarantees. Theoretical results establish sample-complexity bounds: robustness and IF require $\tilde{O}\big(\frac{1}{\epsilon}\sqrt{\log(1/\delta)}\big)$ samples, while group fairness requires $\tilde{O}\big(\frac{1}{\epsilon^2}\log(1/\delta)\big)$, plus a $\tilde{\Omega}\big(\frac{\delta}{\epsilon^2}\big)$ lower bound for GF; NP-hardness of exact Fourier-coefficient computation motivates the approach. Empirically, AFA yields lower estimation error and better sample efficiency than baselines across multiple datasets and model types, demonstrating practical utility for universal, model-agnostic auditing of distributional properties.

Abstract

With the pervasive deployment of Machine Learning (ML) models in real-world applications, verifying and auditing properties of ML models have become a central concern. In this work, we focus on three properties: robustness, individual fairness, and group fairness. We discuss two approaches for auditing ML model properties: estimation with and without reconstruction of the target model under audit. Though the first approach is studied in the literature, the second approach remains unexplored. For this purpose, we develop a new framework that quantifies different properties in terms of the Fourier coefficients of the ML model under audit but does not parametrically reconstruct it. We propose the Active Fourier Auditor (AFA), which queries sample points according to the Fourier coefficients of the ML model, and further estimates the properties. We derive high probability error bounds on AFA's estimates, along with the worst-case lower bounds on the sample complexity to audit them. Numerically we demonstrate on multiple datasets and models that AFA is more accurate and sample-efficient to estimate the properties of interest than the baselines.

Active Fourier Auditor for Estimating Distributional Properties of ML Models

TL;DR

The paper addresses auditing three distributional properties of black-box classifiers—robustness, individual fairness, and group fairness—by avoiding full model reconstruction and instead exploiting a Fourier expansion of the classifier. It introduces the Active Fourier Auditor (AFA), which adaptively queries inputs to recover significant Fourier coefficients via a Goldreich–Levin style procedure and then estimates the properties with PAC guarantees. Theoretical results establish sample-complexity bounds: robustness and IF require samples, while group fairness requires , plus a lower bound for GF; NP-hardness of exact Fourier-coefficient computation motivates the approach. Empirically, AFA yields lower estimation error and better sample efficiency than baselines across multiple datasets and model types, demonstrating practical utility for universal, model-agnostic auditing of distributional properties.

Abstract

With the pervasive deployment of Machine Learning (ML) models in real-world applications, verifying and auditing properties of ML models have become a central concern. In this work, we focus on three properties: robustness, individual fairness, and group fairness. We discuss two approaches for auditing ML model properties: estimation with and without reconstruction of the target model under audit. Though the first approach is studied in the literature, the second approach remains unexplored. For this purpose, we develop a new framework that quantifies different properties in terms of the Fourier coefficients of the ML model under audit but does not parametrically reconstruct it. We propose the Active Fourier Auditor (AFA), which queries sample points according to the Fourier coefficients of the ML model, and further estimates the properties. We derive high probability error bounds on AFA's estimates, along with the worst-case lower bounds on the sample complexity to audit them. Numerically we demonstrate on multiple datasets and models that AFA is more accurate and sample-efficient to estimate the properties of interest than the baselines.

Paper Structure

This paper contains 36 sections, 16 theorems, 107 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Active Fourier Auditor
  4. The Cost of Reconstruction
  5. Model Properties with Fourier Expansion
  6. NP-hardness of Exact Computation
  7. Algorithm: Active Fourier Auditor ()
  8. Algorithmic Insights.
  9. Theoretical Analysis
  10. Upper Bounds on Sample Complexity.
  11. Empirical Performance Analysis
  12. Conclusion and Future Work
  13. Acknowledgements
  14. The cost of auditing with reconstruction: Proof of Proposition 2
  15. Computing Model's Properties with Fourier Coefficients: Proofs of Section 3.2
  16. ...and 21 more sections

Key Result

Proposition 1

There exists a set of orthonormal parity functions $\{\psi_S\}_{S \subseteq [n]}$ such that any function $h: \{-1,1\}^n \to \{-1,1\}$ is decomposed as The Fourier coefficients $\hat{h}(S)\triangleq \E_{\mathbf{x} \sim \mathcal{D}}[h(X) \psi_S(\mathbf{x})]$ are unique for all $S \subseteq [n]$.

Figures (6)

  • Figure 1: A schematic of .
  • Figure 2: begins with the set of all Fourier coefficients, with weight $1$, which is above the threshold $\tau<1$. It proceeds by splitting the bucket and verifies at each level of the tree the weight of the node. If the weight is below the threshold, the algorithm halts. Otherwise, it continues to expand, yielding a set of (informative) trajectories $\Upsilon$, the subsets with large Fourier coefficients are $\{ \cB_{\Upsilon_1}(\cX), \cdots,\cB_{\Upsilon_k}(\cX)\}$.
  • Figure 3: Error (left) and running time (right) of different auditors in estimating statistical parity of COMPAS in LR.
  • Figure 4: Comparison of AFA and random estimator on COMPAS dataset for different values of perturbation parameter $l$.
  • Figure 5: Error (left) and running time (right) of auditors in estimating statistical parity of logistic regression for Student Performance dataset.
  • ...and 1 more figures

Theorems & Definitions (37)

  • Definition 1: Robustness
  • Definition 2: Individual Fairness
  • Definition 3: Statistical Parity
  • Proposition 1: Fourier:Heidari
  • Definition 4: Influence functions
  • Definition 5: PAC-agnostic auditor
  • Proposition 2
  • Definition 6: $\rho$-flipping Influence Function
  • Proposition 3
  • Definition 7: $(\rho,l)$-flipping influence function
  • ...and 27 more