Machine Learning Predictors for Min-Entropy Estimation

TL;DR

This work investigates how machine-learning predictors estimate min-entropy in RNG outputs, focusing on the distinction between average min-entropy and traditional min-entropy under correlated, non-iid data. It develops a theoretical framework based on order-$p$ Markov chains and Generalized Binary Autoregressive Models (gbAR(p)), proving convergence relationships and entropy formulas that link $H_ fty$, $\tilde{H}_\infty$, and their per-bit variants. Through Monte Carlo data generation and experiments with RCNN and GPT-2 predictors, the authors show that ML models tend to estimate the average min-entropy (via modeling conditional probabilities) and can outperform NIST SP 800-90B predictors in certain low-entropy and multi-bit target scenarios, while highlighting the impact of the number of target bits on entropy estimates. The results emphasize the need to consider target-bit counts in entropy assessment for RNGs and suggest that multi-token prediction approaches offer a path to more robust entropy estimation in cryptographic applications, albeit with substantial computational costs in high-entropy regimes.

Abstract

This study investigates the application of machine learning predictors for min-entropy estimation in Random Number Generators (RNGs), a key component in cryptographic applications where accurate entropy assessment is essential for cybersecurity. Our research indicates that these predictors, and indeed any predictor that leverages sequence correlations, primarily estimate average min-entropy, a metric not extensively studied in this context. We explore the relationship between average min-entropy and the traditional min-entropy, focusing on their dependence on the number of target bits being predicted. Utilizing data from Generalized Binary Autoregressive Models, a subset of Markov processes, we demonstrate that machine learning models (including a hybrid of convolutional and recurrent Long Short-Term Memory layers and the transformer-based GPT-2 model) outperform traditional NIST SP 800-90B predictors in certain scenarios. Our findings underscore the importance of considering the number of target bits in min-entropy assessment for RNGs and highlight the potential of machine learning approaches in enhancing entropy estimation techniques for improved cryptographic security.

Figures (5)

• Figure 1: $|\boldsymbol{\alpha}|$ dependance of average min-entropy compared with min-entropy and min-entropy limit for several sequence lengths $n$, correlation scales $p$ and autocorrelation functions (uniform and point-to-point). The data points have been evaluated numerically (see Subsection \ref{['Theoretical min-entropies calculation']}).
• Figure 2: Asymptotic behaviour of average min-entropy and min-entropy per bit in terms of the target space size $n$ for gbAR(p) models with several correlation lengths $p$ and fixed $\beta$. The $\boldsymbol{\alpha}$ arrays are uniform (i.e. all their components are equal). The data points have been evaluated numerically (see Subsection \ref{['Theoretical min-entropies calculation']}).
• Figure 3: Hierarchy of the main models considered in this paper. An arrow from model $M_1$ to model $M_2$ means that every model of type $M_1$ is of type $M_2$.
• Figure 4: Theoretical minimum entropies versus machine learning-based estimations of minimum entropy for gbAR(10) with a uniform $\alpha$ vector and a uniform noise term $\beta = 0.5$ (representing a low entropy scenario). For clarity in visualization, the markers representing different models are slightly offset along the x-axis. Error bars are derived using a binomial proportion confidence interval with a 95% confidence level. The results from the NIST SP 800-90B global predictor tests are emphasized. The highlighted predictor entropies are the minimum of local and global estimates, in this case predominantly influenced by the local estimate. Bitstring predictors try to predict the next bit, while Literal predictors try to predict the next byte. The entropy_non_iid_nist denotes the final outcome of the NIST SP 800-90B analysis, which is the minimum of all conducted tests in the suite. The h_min_limit is the theoretical limit of the min-entropy, interpreted as the min-entropy per bit of the entire process. For this specific gbAR(10) configuration with positive $\alpha$, both min-entropies converge to this limit.
• Figure 5: Comparison of min-entropy estimates: Greedy decoding vs. direct prediction over $n$$target\_bits$ for gbAR(2) models. The discrepancy between the experimental estimate of min-entropy and the theoretical is evident for $|\sqrt{\alpha}|[+1, -1]$, while results align for $|\sqrt{\alpha}|[+1, +1]$.

Theorems & Definitions (37)

• Definition 1
• Remark 2
• Definition 3: cf. ching2006markov, raftery1985model
• Definition 4
• Remark 5
• Lemma 6
• Lemma 7
• Theorem 8: Convergence Theorem bozorgmanesh2016convergence
• Theorem 9
• Definition 10: State-Independent Maximum Transition Probability Order-$p$ Markov Chain