Asymptotic Comparison of ML and MAP Detectors for Multidimensional Constellations

TL;DR

This work addresses the problem of evaluating symbol and bit error probabilities, $P_s(\sigma)$ and $P_b(\sigma)$, for multidimensional constellations on AWGN channels under nonequally likely symbol probabilities. It develops exact closed-form high-SNR asymptotics for arbitrary constellations and input distributions, showing that although MAP decision regions converge to ML regions as $\sigma\to0$, the resulting error costs differ by a constant factor, and the ratio $P^{\text{map}}(\sigma)/P^{\text{ml}}(\sigma)$ tends to a constant $R\le 1$. The union bound is proven to be asymptotically tight for both SEP and BEP, and necessary and sufficient conditions are provided for asymptotic equivalence between MAP and ML detectors. The results extend prior work (e.g., Alvarado12b, Kschischang93) to arbitrary MD constellations and arbitrary input distributions, offering theoretical guidance for probabilistic shaping and labeling design in high-SNR regimes. Practically, the findings quantify the potential performance loss when using ML instead of MAP in nonuniform signaling and inform constellation design and labeling choices to minimize asymptotic BER/SEP.

Abstract

A classical problem in digital communications is to evaluate the symbol error probability (SEP) and bit error probability (BEP) of a multidimensional constellation over an additive white Gaussian noise channel. In this paper, we revisit this problem for nonequally likely symbols and study the asymptotic behavior of the optimal maximum a posteriori (MAP) detector. Exact closed-form asymptotic expressions for SEP and BEP for arbitrary constellations and input distributions are presented. The well-known union bound is proven to be asymptotically tight under general conditions. The performance of the practically relevant maximum likelihood (ML) detector is also analyzed. Although the decision regions with MAP detection converge to the ML regions at high signal-to-noise ratios, the ratio between the MAP and ML detector in terms of both SEP and BEP approach a constant, which depends on the constellation and a priori probabilities. Necessary and sufficient conditions for asymptotic equivalence between the MAP and ML detectors are also presented.

Figures (6)

• Figure 1: System model under consideration. For a given length-$m$ transmitted binary label $\boldsymbol{c}_{i}$, the received vector $\boldsymbol{Y}$ is processed by the MAP or ML detector. The estimated symbol $\hat{\boldsymbol{X}}$ is then converted to an estimated binary label $\hat{\boldsymbol{C}}$.
• Figure 2: ML (dashed blue) and MAP (solid red) decision regions for the constellation in Example \ref{['Example.Regions']} and three values of the noise variance. The area of the constellation points is proportional to their probabilities. As the noise variance decreases, the MAP regions converge to the ML regions.
• Figure 3: Results obtained for the constellations in Example \ref{['example.1D']}: (a) SEP and bounds with MAP and ML detection for the asymmetric constellation, (b) SEP with MAP and ML detection for the symmetric constellation, (c) ratio of SEPs and asymptote for the symmetric constellation given by Corollary \ref{['EP.MAP_vs_ML.Theo']}, and (d) asymptote for the symmetric constellation as a function of the symbol probability $p_{1}$.
• Figure 4: Results obtained for the constellation in Example \ref{['Example.Valenti']}: (a) BEP with MAP and ML detection, and (b) ratio between BEPs in (a) and ${Q}(d/(2\sigma))$.
• Figure 5: Results obtained for the constellation in Example \ref{['example.2D']}: (a) Constellation where the pairs of symbols at MED are marked with solid lines and the symbol probabilities are indicated by the point areas, and (b) asymptotic performance shown as the ratio between SEPs and ${Q}(d/(2\sigma))$.

Theorems & Definitions (8)

• Example 1
• Lemma 1
• Lemma 2
• Theorem 3
• Corollary 4
• Example 2
• Example 3
• Example 4