Stress Testing Machine Learning at $10^{10}$ Scale: A Comprehensive Study of Adversarial Robustness on Algebraically Structured Integer Streams

TL;DR

This work evaluates adversarial robustness of a gradient-boosted classifier on algebraically structured data at scale ($10^{10}$ samples) using Pythagorean triples with the exact relation $a^2 + b^2 = c^2$. It introduces a single-parameter index stream to generate triples efficiently and a Hypothesis-Driven Negative Dataset with nine attack classes, coupled with a fault-tolerant file-index checkpointing pipeline. Results show $99.99\%$ overall accuracy and reveal via SHAP that the model prioritizes the quadratic-generative pattern $b = 2n(n+1)$ (feature $f_7$) over direct arithmetic verification, suggesting learned heuristics can reflect algebraic structure. The study provides a benchmark and methodological toolkit for stress-testing ML on mathematical data, while clearly noting that it does not constitute mathematical verification or universal claims beyond the specific benchmark setting.

Abstract

This paper presents a large-scale stress test of machine learning systems using structured mathematical data as a benchmark. We evaluate the robustness of tree-based classifiers at an unprecedented scale, utilizing ten billion deterministic samples and five billion adversarial counterexamples. Our framework introduces three primary contributions: first, a high-throughput pipeline that reformulates Pythagorean triple generation into a single-parameter index stream, significantly improving computational efficiency over classical methods; second, the Hypothesis-driven Negative Dataset (HND), which categorizes nine classes of adversarial attacks designed to exploit both arithmetic precision and structural patterns; and third, a fault-tolerant infrastructure for reliable large-scale training. Experimental results demonstrate that while LightGBM achieves 99.99% accuracy, feature attribution reveals that the model prioritizes underlying quadratic patterns over direct algebraic verification. These findings suggest that learned heuristics can effectively identify structural representations in numerical data, potentially serving as efficient preprocessors for formal verification methods.