CaFA: Cost-aware, Feasible Attacks With Database Constraints Against Neural Tabular Classifiers

TL;DR

CaFA addresses the gap between feature-space adversarial examples and realizable problem-space attacks on neural tabular classifiers by integrating two data-integrity constraint families—structure constraints and Denial Constraints (DCs)—into a cost-aware attack framework. It introduces TabPGD, a tabular-adapted PGD variant (with CWL0) that respects heterogeneous feature domains and minimizes perturbation costs, followed by SAT/Z3-based projection to ensure DC compliance. Empirical results on three datasets and two model architectures show that DC-based CaFA achieves higher feasible attack success with lower perturbation cost and comparable or faster runtimes than prior methods, and that constraint quality (soundness/completeness) can significantly influence realism and effectiveness. The work demonstrates practical robustness evaluation for deployed tabular models and releases CaFA as an open-source tool, enabling broader, domain-agnostic assessments of adversarial resilience in real-world settings.

Abstract

This work presents CaFA, a system for Cost-aware Feasible Attacks for assessing the robustness of neural tabular classifiers against adversarial examples realizable in the problem space, while minimizing adversaries' effort. To this end, CaFA leverages TabPGD$-$an algorithm we set forth to generate adversarial perturbations suitable for tabular data$-$ and incorporates integrity constraints automatically mined by state-of-the-art database methods. After producing adversarial examples in the feature space via TabPGD, CaFA projects them on the mined constraints, leading, in turn, to better attack realizability. We tested CaFA with three datasets and two architectures and found, among others, that the constraints we use are of higher quality (measured via soundness and completeness) than ones employed in prior work. Moreover, CaFA achieves higher feasible success rates$-$i.e., it generates adversarial examples that are often misclassified while satisfying constraints$-$than prior attacks while simultaneously perturbing few features with lower magnitudes, thus saving effort and improving inconspicuousness. We open-source CaFA, hoping it will serve as a generic system enabling machine-learning engineers to assess their models' robustness against realizable attacks, thus advancing deployed models' trustworthiness.

Figures (14)

• Figure 1: Adversarial examples in the problem space (e.g., a phishing website imitating Google) and the feature space (i.e., feature vectors serving as inputs to ML models). The original feature vector (#1) represents a website correctly detected by ML-based phishing detection. The adversary finds a minimal perturbation (#2) realizable as a problem-space instance while misleading the detector. Attacks, however, may also fail to satisfy data-integrity constraints (#3), rendering them unrealizable in the problem space. Different types of data-integrity constraints exist, including structure (defined by features' domains) and relation (defined by relations between samples and features) constraints.
• Figure 2: Evaluation of constraints' soundness and completeness over the test set. We distinguish Valiant's constraints set from the rest of the DCs, mined by FastADC fastadc. We also mark DCs set with the highest F1 score that we use in most experiments.
• Figure 3: Comparison of the proportion of feasible and misclassified adversarial samples across attacks on MLPs.
• Figure 4: Comparison of the proportion of feasible and misclassified adversarial samples across attacks on TabNets.
• Figure 5: Comparing attacks' costs ($\ell_0$ and standardized-$\ell_\infty$) while accounting for their feasible success (as indicated by the bubble size). Dashed line marks the $\epsilon$ parameter used by CaFA and TabPGD. Measures were averaged over all datasets (see App. \ref{['app:addexps:cost']} for per-dataset results). Larger bubbles closer to the bottom-left corner are better.