Large and Small Deviations for Statistical Sequence Matching

TL;DR

This work studies statistical sequence matching across two databases when the generating distributions are unknown, employing a universal generalized likelihood ratio test (GLRT) based on the generalized Jensen-Shannon divergence. It provides explicit large-deviation and small-deviation (finite-blocklength) characterizations of the mismatch, false-reject, and false-alarm probabilities, proving the optimality of Unnikrishnan’s GLRT under the generalized Neyman-Pearson criterion for known match counts and extending the analysis to unknown counts via a two-phase testing approach. The results recover Gutman’s classical large-deviation exponents for multi-hypothesis classification in the special case of a single match, while strengthening small-deviation results by removing unnecessary distribution-uniqueness assumptions. The paper also delivers nonasymptotic second-order bounds via Berry-Esseen-type analyses and validates the theory with numerical examples, highlighting practical tradeoffs between detection sensitivity and error control in finite samples. Finally, it generalizes to the unknown-number-of-matches setting, introducing a false-alarm metric and discussing the limitations and challenges in achieving distribution-free mismatch exponents, with connections to sequential and reduced-complexity classification problems.

Abstract

We revisit the problem of statistical sequence matching between two databases of sequences initiated by Unnikrishnan (TIT 2015) and derive theoretical performance guarantees for the generalized likelihood ratio test (GLRT). We first consider the case where the number of matched pairs of sequences between the databases is known. In this case, the task is to accurately find the matched pairs of sequences among all possible matches between the sequences in the two databases. We analyze the performance of the GLRT by Unnikrishnan and explicitly characterize the tradeoff between the mismatch and false reject probabilities under each hypothesis in both large and small deviations regimes. Furthermore, we demonstrate the optimality of Unnikrishnan's GLRT test under the generalized Neyman-Person criterion for both regimes and illustrate our theoretical results via numerical examples. Subsequently, we generalize our achievability analyses to the case where the number of matched pairs is unknown, and an additional error probability needs to be considered. When one of the two databases contains a single sequence, the problem of statistical sequence matching specializes to the problem of multiple classification introduced by Gutman (TIT 1989). For this special case, our result for the small deviations regime strengthens previous result of Zhou, Tan and Motani (Information and Inference 2020) by removing unnecessary conditions on the generating distributions.

Figures (6)

• Figure 1: Plot of $F_l(P^{M_1},Q^{M_2},\alpha,\lambda,K)$ as a function of $\lambda$ when $\alpha=2$, $M_1=4$, $M_2=K=2$, $(P_1,P_2,P_3,P_4)=\mathrm{Bern}(0.1,0.2,0.3,0.4)$ and $Q_1=P_1$, $Q_2=P_2$.
• Figure 2: Plot of $\chi_l^*(n,\varepsilon|P^{M_1},Q^{M_2},K,\alpha)$ as a function of $n$ for various values of $\varepsilon$ when $\alpha=2$, $M_1=4$, $M_2=K=2$, $(P_1,P_2,P_3,P_4)=\mathrm{Bern}(0.1,0.2,0.3,0.4)$ and $Q_1=P_1$, $Q_2=P_2$. In this case, $\tau_l=1$. Thus, the multivariate Gaussian cdf degenerates to the univariate Gaussian cdf.
• Figure 3: Plot of $\chi_l^*(n,\varepsilon|P^{M_1},Q^{M_2},K,\alpha)$ as a function of $n$ for various values of $\varepsilon$ when $\alpha=2$, $M_1=4$, $M_2=K=1$, $(P_1,P_2,P_3,P_4)=\mathrm{Bern}(0.1,0.2,0.327,0.4)$ and $Q=P_1$. In this case, $\tau_l=2$ and the multivariate Gaussian cdf is required to obtain the second-order expansion $\chi_l^*(n,\varepsilon|P^{M_1},Q^{M_2},K,\alpha)$.
• Figure 4: Simulated exponents for the maximal mismatch probabilities of Unnikrishnan's test and the simple test with threshold $\lambda=10^{-4}$ when $M_1=4$, $M_2=K=2$ under any tuple of generating distributions $(P^{M_1},Q^{M_2})\in\mathcal{Q}$. The error bar denotes two standard deviations below and above the mean value. The plot empirically confirms the mismatch exponents of both tests converge to the asymptotic lower bounded $\lambda$ claimed in Theorems \ref{['ld:known']} and \ref{['simple:same:mismatch']} as the sample size $n$ increases. On the right hand side, the simulated false reject exponents of both tests are plotted.
• Figure 5: Simulated false reject exponents of Unnikrishnan's test and the simple test with threshold $\lambda=10^{-4}$ when $M_1=4$, $M_2=K=2$ for generating distributions $P_1=Q_1=\mathrm{Bern}(0.1)$, $P_2=Q_2=\mathrm{Bern}(0.11)$, $P_3=\mathrm{Bern}(0.12)$ and $P_4=\mathrm{Bern(0.13)}$. The error bar denotes two standard deviations below and above the mean value. As observed, the plots empirically confirms that the false reject exponent of Unnikrishnan's test converges towards the theoretical bound in Theorem \ref{['ld:known']}.

Theorems & Definitions (12)

• Theorem 1
• Lemma 1
• Theorem 2
• Theorem 3
• Lemma 2
• Theorem 4
• Theorem 5
• Lemma 3
• Lemma 4
• proof