Machine Learning needs Better Randomness Standards: Randomised Smoothing and PRNG-based attacks

TL;DR

This work reveals that randomness used in ML, especially within Randomised Smoothing, can be covertly manipulated via backdoored PRNGs, leading to miscalibrated robustness certificates. It introduces two attack classes—a naive noise-distribution swap and a bit-flipping PRNG attack—that can inflate or deflate the certified radius by large factors (up to $81\times$) while evading standard NIST tests. The paper demonstrates that existing randomness standards are insufficient for ML safety-critical settings and argues for ML-specific randomness guarantees and testing, including normality-focused diagnostics. The findings have practical implications for security and privacy in ML systems and motivate a shift toward more comprehensive, domain-aware randomness standards and defenses in ML toolchains.

Abstract

Randomness supports many critical functions in the field of machine learning (ML) including optimisation, data selection, privacy, and security. ML systems outsource the task of generating or harvesting randomness to the compiler, the cloud service provider or elsewhere in the toolchain. Yet there is a long history of attackers exploiting poor randomness, or even creating it -- as when the NSA put backdoors in random number generators to break cryptography. In this paper we consider whether attackers can compromise an ML system using only the randomness on which they commonly rely. We focus our effort on Randomised Smoothing, a popular approach to train certifiably robust models, and to certify specific input datapoints of an arbitrary model. We choose Randomised Smoothing since it is used for both security and safety -- to counteract adversarial examples and quantify uncertainty respectively. Under the hood, it relies on sampling Gaussian noise to explore the volume around a data point to certify that a model is not vulnerable to adversarial examples. We demonstrate an entirely novel attack, where an attacker backdoors the supplied randomness to falsely certify either an overestimate or an underestimate of robustness for up to 81 times. We demonstrate that such attacks are possible, that they require very small changes to randomness to succeed, and that they are hard to detect. As an example, we hide an attack in the random number generator and show that the randomness tests suggested by NIST fail to detect it. We advocate updating the NIST guidelines on random number testing to make them more appropriate for safety-critical and security-critical machine-learning applications.

Figures (6)

• Figure 1: A visual example of how manipulated randomness affects Randomised Smoothing. Here we aim to certify a point in the middle. To do that we sample data around it using noise -- a normal noise sampling for Randomised Smoothing is shown in black. Manipulated random noise with lower kurtosis, skewness and higher kurtosis are shown in order from left to right. Manipulated noise misrepresents the true decision space and leads to incorrect prediction confidence.
• Figure 2: Sampling from a Gaussian distribution constructed using Uniform distributions with the Ziggurat Method marsaglia_ziggurat_2000.
• Figure 3: These figures show the sampled probability densities against the normal pdf in orange for all considered values of the tunable parameters for each of the bit-flipping PRNG attacks proposed. The negative-kurtosis, skewness and positive-kurtosis attacks all alter the sampled distribution in a self-explanatory manner.
• Figure 4: These figures show the NIST test pass rates for the frequency, cumulative sums and runs tests, the three tests that managed to detect all the attacked PRNGs with high confidence at an input size of $10^6$. These plots show how the pass rates vary as the input size is increased from the minimum recommended value by NIST, $10^2$, to $10^6$ for the two attacks that were the hardest to detect: the skewness attack with $\beta=4$ and the positive-kurtosis attack with $\gamma=4$. The attacks become detectable only after the input size is increased to $> 10^5$.
• Figure 5: These figures show the quantile-quantile (QQ) plots for $10^5$ samples collected from PRNGs subject to the negative-kurtosis, skewness and positive-kurtosis attacks, showing the difference in CDF relative to the normal distribution. While attacks that significantly alter the distribution can be detected visually, this is not always apparent, as is the case with the skewness attack with $\beta=4$ and positive-kurtosis attacks with $\gamma=2$ and $\gamma=4$.