Typical reconstruction limit and phase transition of maximum entropy method

TL;DR

This work analyzes the typical performance of the Maximum Entropy Method (MEM) for underdetermined, binary signal reconstruction using the replica method in the $N\to\infty$ limit with compression ratio $\alpha = M/N$. By introducing two default-model deviation schemes—the continuous deviation $\varepsilon$ and the discrete flip ratio $\eta$—the authors derive phase diagrams showing sharp transitions between successful and failed MEM reconstructions, dependent on sparsity $\rho$ and model mismatch. They validate the analytic predictions with ADMM-based numerical reconstructions, finding good agreement and revealing that MEM can underperform compared to $L_1$-norm regularization even in dense regimes. The results emphasize the critical role of the default model in MEM’s reliability and motivate careful evaluation of default-model choices across domains and applications.

Abstract

We investigate the dependence of the maximum entropy method (MEM) reconstruction performance on the default model. The maximum entropy method is a reconstruction technique that utilizes prior information, referred to as the default model, to recover original signals from observed data, and it is widely used in underdetermined systems. The broad applications have been reported in fields ranging from the analysis of observational data in seismology and astronomy, to large-scale computations in quantum chemistry, and even in social sciences such as linguistics. However, a known drawback of MEM is that its results depend on the assumed default model. In this study, we employ the replica method to elucidate how discrepancies in the default model affect the reconstruction of signals with specific distributions. We report that in certain cases, even small discrepancies can induce phase transitions, leading to reconstruction failure. Additionally, by comparing MEM with reconstruction based on L1-norm optimization, a method proposed in recent years, we demonstrate that MEM exhibits lower reconstruction accuracy under certain conditions.

Figures (5)

• Figure 1: The phase diagram of the deviation model. (a)The result of the replica analysis and (b)one of ADMM calculation for $\rho=0.2$. (c)The result of the replica analysis and (d)one of ADMM calculation for $\rho=0.8$.
• Figure 2: The phase diagram of the flipping model. (a)The result of the replica analysis and (b)one of ADMM calculation for $\rho=0.2$. (c)The result of the replica analysis and (d)one of ADMM calculation for $\rho=0.8$.
• Figure 3: Illustration of the impact of flipping in the default model on the estimated signal. (a)When the original signal is $\bm{x}^0 = (1,0)$ and the default model perfectly matches it; $\bm{w} = (1,1)$, the reconstructed values $\bm{x}_{\text{est}}$ closely align with $\bm{x}^0$. (b) When the default model is flipped as $\bm{w} = (0,0)$, the estimated values shift significantly, demonstrating the influence of default model discrepancies on MEM reconstruction.
• Figure 4: The $\rho$-dependence of MSE. (a) and (b) show dependece about $\alpha$ and $\varepsilon$ of the deviation model. (c) and (b) show dependece about $\alpha$ and $\eta$ of the flipping model. Even though there is no transition point at $\varepsilon$ derection of the deviation model, the point cause at $\eta$ of the deviation model. This is because the flipping model keeps discrete variables at each set up.
• Figure 5: The phase diagram of (a)$L_1$-norm regularization, (b) $\sim$ (e)MEM estimation of the flipping model with $\eta = 0$, $0.05$, $0.1$, $0.2$. Even though the flipping model with $\eta = 0$ shown in (b) is represent the original signal perfectly, the reconstruction with the finite $\eta$ causes the failure phase in (c) $\sim$ (e). (f) $\sim$ (i)MEM estimation of the deviation model with $\varepsilon = 0.2$, $0.4$, $0.6$, $0.8$. When the original signal is fully dense, i.e. $\\rho = 1$, the original signal and the default model coincide completely, and perfect reconstruction becomes possible. The horizontal and vertical axis of every figure denote the sparsity $\\rho$ and $\\alpha$.