Energy and polarization based on-line interference mitigation in radio interferometry

TL;DR

This work tackles real-time mitigation of radio-frequency interference in post-correlation radio interferometry by fusing energy-based spectral kurtosis with polarization directional statistics to flag RFI within small windows of size $W\approx 20$. To satisfy real-time constraints, it introduces reinforcement learning to adaptively select mixed-precision arithmetic across 14 operation groups, achieving near double-precision detection while reducing computational cost on GPUs. Results from both synthetic tests and LOFAR-like real data demonstrate robust detection of transient RFI and substantial data savings, with online performance closely matching offline baselines. The approach is poised for deployment in the LOFAR correlator, aided by public release of the source code and the potential for online threshold adaptation and downstream beamforming integration.

Abstract

Radio frequency interference (RFI) is a persistent contaminant in terrestrial radio astronomy. While new radio interferometers are becoming operational, novel sources of RFI are also emerging. In order to strengthen the mitigation of RFI in modern radio interferometers, we propose an on-line RFI mitigation scheme that can be run in the correlator of such interferometers. We combine statistics based on the energy as well as the polarization alignment of the correlated signal to develop an on-line RFI mitigation scheme that can be applied to a data stream produced by the correlator in real-time, especially targeted at low duty-cycle or transient RFI detection. In order to improve the computational efficiency, we explore the use of both single precision and half precision floating point operations in implementing the RFI mitigation algorithm. This ideally suits its deployment in accelerator computing devices such as graphics processing units (GPUs) as used by the LOFAR correlator. We provide results based on real data to demonstrate the efficacy of the proposed method.

Figures (5)

• Figure 1: The reward for $100 000$ episodes, each episode has $100$ steps. The ensemble has $4$ models and each model is randomly initialized and trained using different random data.
• Figure 2: Spectrograms of simulated data (amplitude) of 4 correlations, (a) data before RFI mitigation and data after RFI mitigation using (b) spectral kurtosis (c) directional statistics, and (d) both methods. The false alarm and missed detection probabilities are (b) 0.01, 0.89 (c) 0.01, 0.06 and (d) 0.01, 0.05. The dark colours denote points that are considered to be RFI. The full spectrogram is of size $2000\times 512$ time-frequency samples covering 1 s and 200 kHz. The RFI mitigation is performed by using a time-frequency window of size $10\times 2$ ($W=20$) which is much smaller than the full data window size. The INR is $100$.
• Figure 3: The false alarm and missed detection probabilities, averaged over $400$ Monte Carlo runs. The results for RFI with randomly varying polarization are shown in (a) and for unpolarized RFI are shown in (b). Results are using SK: spectral kurtosis, DS: directional statistics, and BOTH: both aforementioned methods. The hyperparameters for both methods are determined by providing a desired false alarm probability of $0.05$ as discussed in section \ref{['ssec:hyper']} with window size $W=20$. The hyperparameters are kept constant for all values of the INR. We see that the SK method performs poorly for low values of the INR (i.e., when the RFI signals are weak), suggesting that the hyperparameters for the SK method needs fine-tuning for each value of the INR for possible improvement. In contrast, the DS method is more robust to the varying INR.
• Figure 4: Spectrogram of one baseline, Stokes I amplitude at 14 MHz, (a) data (b) result after off-line flagging aoflagger (c) result after flagging using directional statistics of polarization as in Table \ref{['InsTrainedDS']} (d) result after flagging using spectral kurtosis as in Table \ref{['InsTrainedSK']} and (e) result after flagging using both directional statistics of polarization and spectral kurtosis. The off-line flagging uses the full time-frequency domain of size $3500\times 64$ (1 hour $\times$ 200 kHz) while the online flagging only uses time-frequency windows of size $10\times 2$. The thresholds used for the algorithms are determined as described in section \ref{['ssec:hyper']} by providing a false alarm probability of $0.05$ as the input. We see that the off-line method is able to flag more data (also by the dilation of the flag mask) than the online methods but the objective of the online method is to flag data before any averaging is being performed on the data and the above example does not represent its intended purpose.
• Figure 5: Spectrograms of one baseline, Stokes I amplitude at 14 MHz, (a) data (b) result after online RFI mitigation with 32 bit operations (c) result after RFI mitigation with mixed precision operations as outlined in Tables \ref{['InsTrainedSK']} and \ref{['InsTrainedDS']}. The difference in the data that are flagged as RFI between (b) and (c) is less than 1%. The hyperparameters (including the window size $W$) used for the algorithms are similar to the ones used in Fig. \ref{['fig:lofar_lba1']}. The time-frequency window corresponds to about 12 hours $\times$ 200 kHz.