Evaluation Metrics and Methods for Generative Models in the Wireless PHY Layer

TL;DR

The paper tackles the lack of physically interpretable metrics for evaluating generative models of wireless PHY channels. It introduces an application-motivated framework with three core routines—spectral efficiency analysis, codebook fingerprinting, and application cross-check—grounded in real measurement data and applicable across GMM, VAE, DM, and GAN models. The findings show that relying solely on traditional ML metrics like MMD can be misleading, while the proposed metrics provide consistent, explainable insights; GMMs often reproduce both channel power and directional information most faithfully in the tested scenarios. The work delivers practical evaluation tools for designing and selecting generative models for wireless channels, with implications for downstream ML tasks in channel estimation and compression and a roadmap for extending the framework to broader PHY contexts.

Abstract

Generative models are typically evaluated by direct inspection of their generated samples, e.g., by visual inspection in the case of images. Further evaluation metrics like the Fréchet inception distance or maximum mean discrepancy are intricate to interpret and lack physical motivation. These observations make evaluating generative models in the wireless PHY layer non-trivial. This work establishes a framework consisting of evaluation metrics and methods for generative models applied to the wireless PHY layer. The proposed metrics and methods are motivated by wireless applications, facilitating interpretation and understandability for the wireless community. In particular, we propose a spectral efficiency analysis for validating the generated channel norms and a codebook fingerprinting method to validate the generated channel directions. Moreover, we propose an application cross-check to evaluate the generative model's samples for training machine learning-based models in relevant downstream tasks. Our analysis is based on real-world measurement data and includes the Gaussian mixture model, variational autoencoder, diffusion model, and generative adversarial network as generative models. Our results under a fair comparison in terms of model architecture indicate that solely relying on metrics like the maximum mean discrepancy produces insufficient evaluation outcomes. In contrast, the proposed metrics and methods exhibit consistent and explainable behavior.

Figures (9)

• Figure 1: Structure of a VAE with cg distributions for $q_{{\bm{\phi}}}({\bm{z}}{\,|\,}{\bm{h}})$ and $p_{{\bm{\theta}}}({\bm{h}}{\,|\,}{\bm{z}})$. The encoder and decoder each represent a nn.
• Figure 2: Markov chain of the DM involving the forward process with $q({\bm{h}}_{t}{\,|\,} {\bm{h}}_{t-1})$ and the approximated reverse process with $p_{\bm{\theta}}({\bm{h}}_{t-1}{\,|\,} {\bm{h}}_{t})$.
• Figure 3: Illustration of a GAN with generator $G_{\bm{\theta}}({\bm{z}})$ and discriminator $D_{\bm{\zeta}}({\bm{h}})$ each representing a nn.
• Figure 4: Illustration of the application cross-check.
• Figure 5: Illustration showing the measurement site with the BS at a rooftop and LOS/NLOS conditions for the mt locations Hellings2019.