SPATA: Systematic Pattern Analysis for Detailed and Transparent Data Cards

TL;DR

The paper tackles robustness evaluation when access to private training data is restricted. It introduces SPATA, a deterministic projection method that discretizes each feature into domain-specific subdomains and builds a domain-independent representation of data instances via recursive bin mappings. SPATA enables constructing data cards and conducting adversarial robustness analyses without exposing original data, using an open-source Python/C implementation and exporting patterns as JSON/CSV. Empirical validation on CICIDS and IoT23 datasets shows comparable generalization and robustness between models trained on original versus SPATA-projected data, with SHAP-based feature importances largely preserved, supporting its usefulness for privacy-preserving external verification.

Abstract

Due to the susceptibility of Artificial Intelligence (AI) to data perturbations and adversarial examples, it is crucial to perform a thorough robustness evaluation before any Machine Learning (ML) model is deployed. However, examining a model's decision boundaries and identifying potential vulnerabilities typically requires access to the training and testing datasets, which may pose risks to data privacy and confidentiality. To improve transparency in organizations that handle confidential data or manage critical infrastructure, it is essential to allow external verification and validation of AI without the disclosure of private datasets. This paper presents Systematic Pattern Analysis (SPATA), a deterministic method that converts any tabular dataset to a domain-independent representation of its statistical patterns, to provide more detailed and transparent data cards. SPATA computes the projection of each data instance into a discrete space where they can be analyzed and compared, without risking data leakage. These projected datasets can be reliably used for the evaluation of how different features affect ML model robustness and for the generation of interpretable explanations of their behavior, contributing to more trustworthy AI.

Figures (6)

• Figure 1: Subdomains of features with an equivalent distribution of values.
• Figure 2: Subdomains of features with different distributions of values.
• Figure 3: Recursive mapping of feature subdomains to create a code.
• Figure 4: Parallel mapping of multiple features to create a combination.
• Figure 5: Overlaps of feature codes and combinations.