Statistical isotropy of the universe and the look-elsewhere effect

TL;DR

The paper critiques a recent claim of strong CMB statistical anisotropy by Jones et al., arguing that two of the four tested anomalies do not probe anisotropy and that look-elsewhere effects significantly dilute the reported significance. It derives the distribution of the product of the four smallest $p$-values among $n_T$ tests, showing that the observed joint $p$-value can arise with a moderate number of tests and is not as extreme as claimed. By estimating test correlations, the authors deduce that 16–50 independent tests could plausibly reduce the joint significance to $3\sigma$ or $2\sigma$, respectively, casting doubt on the anisotropy claim. They compile a list of anomaly tests and propose generalizations, concluding that current data are consistent with $\Lambda$CDM and statistical isotropy, with publication bias and parameter choices likely increasing the number of performed tests.

Abstract

Recently, Jones et al. [arXiv:2310.12859] claimed strong evidence for the statistical anisotropy of the universe. The claim is based on a joint analysis of four different anomaly tests of the cosmic microwave background data, each of which is known to be anomalous, with a lower level of significance. They reported a combined $p$-value of about $3\times 10^{-8}$, which is more than a $5σ$ level of significance. We observe that statistical anisotropy is not even relevant for two of the four considered tests, which seems sufficient to invalidate the authors' claim. Furthermore, even if one reinterprets the claim as evidence against $Λ$CDM rather than statistical anisotropy, we argue that this result significantly suffers from the look-elsewhere effect. Assuming a set of independent (i.e., uncorrelated) tests, we show that if the four tests with the smallest $p$-values are cherry-picked from 10 independent tests, the $p$-value reported by Jones et al. corresponds to only $3σ$ significance. If there are 27 independent tests, the significance falls to $2σ$. These numbers, however, overstate our argument, since the four tests used by Jones et al. are slightly correlated. Determining the correlation of Jones et al.'s tests by comparing their joint $p$-value with the product of the four separate $p$-values, we find that about 16 or 50 tests are sufficient to reduce the significance of Jones et al.'s results to 3$σ$ or 2$σ$ significance, respectively. We also provide a list of anomaly tests discussed in the literature (and propose a few generalizations), suggesting that very plausibly 16 (or even 50) independent tests have been published, and possibly many more have been considered but not published. We conclude that the current data is consistent with the $Λ$CDM model and, in particular, with statistical isotropy.

Figures (6)

• Figure 1: The probability distribution ${\cal{P}}_4(x)$ for $n_T=100$.
• Figure 2: Plot of $x {\cal{P}}_4(x)$ for $n_T=100$. $x {\cal{P}}_4(x)$ is the probability density for $\ln x$. Since $x$ is shown on a logarithmic scale, the area under this curve is proportional to the probability that $x$ lies in a given range. The red vertical dashed line shows $x = x_{\text{JCSA}}$, the value of $x$ found by JCSA, which can be seen to be in the region of high probability. Numerical integration shows that for this case, there is a probability of 33.8% that $x$ will be smaller than $x_{\text{JCSA}}$.
• Figure 3: The median of ${\cal{P}}_4$ as a function of $n_T$ and its comparison with the result of the Monte Carlo simulation. For $n_T=133$, we obtain $x \simeq x_{\text{JCSA}}$ as the median of ${\cal{P}}_4(x)$.
• Figure 4: The measured value of $x$ with $2\sigma$ significance, $x_{2\sigma}$, according to the probability distribution ${\cal{P}}_4$, as a function of $n_T$. The $2\sigma$ significance of $x= x_{\text{JCSA}}$ requires $n_T \simeq 27$.
• Figure 5: The statistical significance $\alpha$ (in units of $\sigma$) when the measured value is $x = x_{\text{JCSA}}$, as a function of $n_T$, according to the probability distribution ${\cal{P}}_4$. The measurement of $x= x_{\text{JCSA}}$ corresponds to roughly $2\sigma$ significance if $n_T = 27$ and to $3\sigma$ significance if $n_T = 10$.