Statistical Analysis of Data Repeatability Measures

Abstract

The advent of modern data collection and processing techniques has seen the size, scale, and complexity of data grow exponentially. A seminal step in leveraging these rich datasets for downstream inference is understanding the characteristics of the data which are repeatable -- the aspects of the data that are able to be identified under a duplicated analysis. Conflictingly, the utility of traditional repeatability measures, such as the intraclass correlation coefficient, under these settings is limited. In recent work, novel data repeatability measures have been introduced in the context where a set of subjects are measured twice or more, including: fingerprinting, rank sums, and generalizations of the intraclass correlation coefficient. However, the relationships between, and the best practices among these measures remains largely unknown. In this manuscript, we formalize a novel repeatability measure, discriminability. We show that it is deterministically linked with the correlation coefficient under univariate random effect models, and has desired property of optimal accuracy for inferential tasks using multivariate measurements. Additionally, we overview and systematically compare repeatability statistics using both theoretical results and simulations. We show that the rank sum statistic is deterministically linked to a consistent estimator of discriminability. The power of permutation tests derived from these measures are compared numerically under Gaussian and non-Gaussian settings, with and without simulated batch effects. Motivated by both theoretical and empirical results, we provide methodological recommendations for each benchmark setting to serve as a resource for future analyses. We believe these recommendations will play an important role towards improving repeatability in fields such as functional magnetic resonance imaging, genomics, pharmacology, and more.

Figures (7)

• Figure 1: The relation between discriminability and ICC under the ANOVA model with Gaussian random effects. See Section \ref{['sec:anova_approx']}.
• Figure 1: ANOVA simulations when the Gaussian assumption is satisfied (left) or violated with logarithm transformations (right). Simulated distributions of estimators are plotted on the top, including the discriminability estimation (using the estimator $\Tilde{D}$ or the rank sum version $\hat{D}_{rs}$), the fingerprint index estimation, and the ICC estimation. Simulated permutation test powers are plotted on the bottom, where solid lines and dotted lines represent nonparametric and parametric statistics, respectively. $\sigma^2 = 5$. $\sigma_\mu^2 = 3$. $n$ ranges from $5$ to $40$. $1$,$000$ iterations in total. See Section \ref{['sec:anova']}.
• Figure 1: Comparison of data repeatability measures.
• Figure 2: Non-decreasing bounds of the discriminability approximation (\ref{['eq:approxF']}) using functions of I2C2 under the MANOVA model with random Gaussian effects. The dispersion measures, defined as $V_1^2/W_1$ and $V_2^2/W_2$, are fixed at $10$ or $30$. The upper (red) and lower (blue) bounds are color coded, respectively. The dispersion $10$ scenario is plotted with solid lines whereas the dispersion $30$ scenario is plotted with dashed lines. $V_1, W_1$ (or $V_2, W_2$) are the sum and the sum of squares of the positive (or negative) eigenvalues from the distributional decomposition (\ref{['eq:decomposition']}). See Section \ref{['sec:anova_approx']} .
• Figure 2: MANOVA simulations when the Gaussian assumption is satisfied (left) or violated with element-wise logarithm transformations (right). Simulated distributions of estimators are plotted on the top, including the discriminability estimation (using the estimator $\Tilde{D}$ or the rank sum version $\hat{D}_{rs}$), the fingerprint index, and the I2C2. Simulated permutation test powers are plotted on the bottom, where solid lines and dotted lines represent nonparametric and parametric statistics, respectively. $\sigma^2 = 5$. $\sigma_\mu^2 = 3, \rho = 0.5, l = 10$. $n$ ranges from $5$ to $40$. $1$,$000$ iterations in total. See Section \ref{['sec:manova']}.

Theorems & Definitions (8)

• Lemma A.1: local discriminability is unbiased for discriminability
• Proof 1
• Lemma A.2: Unbiasedness of Sample Discriminability
• Proof 2
• Lemma A.3: Consistency of Sample Discriminability
• Proof 3
• Lemma D.1: Monotonicity of Sums of Positive or Negative Eigenvalues
• Proof 4