Out-of-Distribution Radar Detection with Complex VAEs: Theory, Whitening, and ANMF Fusion

TL;DR

This paper tackles the problem of detecting weak complex-valued radar targets in non-Gaussian, range-varying clutter. It proposes a complex-valued variational autoencoder (CVAE) trained only on clutter-plus-noise to perform out-of-distribution detection, operating directly on complex samples to preserve phase and Doppler structure, and it introduces local whitening and a fusion strategy with the ANMF detector. Key contributions include a complex latent-VAE with a closed-form KL divergence, a practical local whitening scheme, and a decision-level, PT-based fusion rule that maintains CFAR under dependence; these components are validated on simulated correlated Gaussian and compound-Gaussian clutter as well as real CSIR sea-clutter data. The results show that whitening enhances CVAE performance, that CVAE can match or exceed traditional detectors in non-Gaussian settings, and that the CVAE–ANMF fusion provides robust, CFAR-compliant detection across Doppler bins, representing a flexible alternative to model-based radar detectors.

Abstract

We investigate the detection of weak complex-valued signals immersed in non-Gaussian, range-varying interference, with emphasis on maritime radar scenarios. The proposed methodology exploits a Complex-valued Variational AutoEncoder (CVAE) trained exclusively on clutter-plus-noise to perform Out-Of-Distribution detection. By operating directly on in-phase / quadrature samples, the CVAE preserves phase and Doppler structure and is assessed in two configurations: (i) using unprocessed range profiles and (ii) after local whitening, where per-range covariance estimates are obtained from neighboring profiles. Using extensive simulations together with real sea-clutter data from the CSIR maritime dataset, we benchmark performance against classical and adaptive detectors (MF, NMF, AMF-SCM, ANMF-SCM, ANMF-Tyler). In both configurations, the CVAE yields a higher detection probability Pd at matched false-alarm rate Pfa, with the most notable improvements observed under whitening. We further integrate the CVAE with the ANMF through a weighted log-p fusion rule at the decision level, attaining enhanced robustness in strongly non-Gaussian clutter and enabling empirically calibrated Pfa control under H0. Overall, the results demonstrate that statistical normalization combined with complex-valued generative modeling substantively improves detection in realistic sea-clutter conditions, and that the fused CVAE-ANMF scheme constitutes a competitive alternative to established model-based detectors.

Figures (12)

• Figure 1: Complex-valued VAE architecture used in this work. The encoder applies complex 1D convolutions and pooling to a Doppler profile $\mathbf{z}\in\mathbb{C}^m$ and produces three heads $(\boldsymbol{\mu},\tilde{\boldsymbol{s}},\tilde{\boldsymbol{\delta}})$. The unconstrained outputs $\tilde{\boldsymbol{s}}$ and $\tilde{\boldsymbol{\delta}}$ are mapped to the variance $\boldsymbol{\sigma}$ and pseudo-variance $\boldsymbol{\delta}$ via a softplus and a complex softsign$\cdot\boldsymbol{\sigma}$ transform. A complex reparameterization with two real noise vectors $\boldsymbol{\varepsilon}_r,\boldsymbol{\varepsilon}_i$ yields latent samples $\mathbf{x}\in\mathbb{C}^q$, which are decoded back to $\hat{\mathbf{z}}\in\mathbb{C}^m$ via complex transposed convolutions and upsampling.
• Figure 2: Two-branch detection pipeline combining data-driven and model-based approaches.
• Figure 3: CSIR sea-clutter datasets used in Section \ref{['sec:results']}. Left: range-time maps (96 range gates, 15 m resolution). Right: Doppler-time spectrograms at the gates used for detection analysis. Same color scale across panels.
• Figure 4: $P_d$ vs. SNR at $d=0$ under different simulated noise models ($P_{fa}=10^{-2}$, $\rho=0.5$, $\mu=1$, $m=16$, $K=32$).
• Figure 5: $P_d$-SNR-Doppler maps for the cGN+AWGN setting ($P_{fa}=10^{-2}$). Left to right: AMF-SCM, ANMF-Tyler, CVAE (raw), CVAE (whitened), and oracle CVAE whitening.