Evaluating the effectiveness of radio frequency interference removal algorithms for single pulse searches

TL;DR

Radio Frequency Interference (RFI) poses a major challenge for detecting single-pulse phenomena like pulsars, RRATs, and FRBs in real-time transient pipelines. The authors introduce a controlled evaluation framework that injects model pulses into synthetic, SKA-Mid-like dynamic spectra across varied RFI environments, then applies selected RFIM algorithms (SKF, IQRM, ZDMF) individually and in combinations, followed by a realistic single-pulse search to measure pulse recovery. They find that no single method suffices: combining IQRM with ZDMF (and, in some cases, SKF with ZDMF) generally improves pulse recovery, though ZDMF can degrade low-$DM$ recoveries and residual RFI can still trigger false positives or timeouts. The work provides actionable guidance for RFIM choice in real-time surveys and a flexible framework to benchmark future mitigation approaches for next-generation radio telescopes.

Abstract

Radio Frequency Interference (RFI), the presence of artificial and/or terrestrial signals in astronomical data, poses a great challenge to the search for pulsars and radio transients, such as Rotating Radio Transients (RRATs) and Fast Radio Bursts (FRBs), by obscuring or distorting the signal of interest and resulting in large numbers of erroneous detections. RFI mitigation algorithms aim to remove this interference and improve the chance of detection of transients, but with the growing number of techniques, selecting the most appropriate method for a given survey can be problematic. The choice of method is particularly important in real-time searches planned for next-generation telescopes such as those of the SKAO, where there is no possibility to reprocess the data. In this paper, we explore the algorithm selection problem by injecting pulses into data which simulates several RFI environments. A set of these files is then cleaned using RFI mitigation algorithms and run through a single pulse search pipeline to analyse the recovery of the injected pulses. We examine the recovery of the injected single pulses with an emphasis on a number of cases spanning a range of pulse brightness, width and dispersion measure. The efficacy and side effects of a few popular RFI excision methods, namely IQRM, SKF, and ZDMF are evaluated.

Figures (38)

• Figure 1: Data in a filterbank file that contains a simulated radio transient with a DM of $500\,\mathrm{pc/cm^2}$ with a S/N of 141 and added RFI. This test vector shows strong RFI ($\mathrm{2\sigma}$) of all the types, i.e. narrowband constant, narrowband periodic and broadband periodic RFI. A Gaussian-shaped, dispersed pulse can be seen sweeping from around $1.11\,\mathrm{s}$ in time in the highest frequency channel ($1670\,\mathrm{MHz}$) up to $1.5\,\mathrm{s}$ in the lowest frequency channel ($1350\,\mathrm{MHz}$). The structures around $\mathrm{1606~MHz}$ and $\mathrm{1410~MHz}$ mimic the constant narrowband RFI that can be seen from fixed-frequency terrestrial transmitters, which affect the entire observation. The periodic structures affecting the full bandwidth around $1.5 ~\mathrm{s}$ in time are similar to those of lightning or some other periodic broadband signal. The periodic structures affecting a narrowband of frequencies from $\mathrm{0.22~s}$ and $\mathrm{0.7~s}$ are like those produced by a communication satellite or mobile phones. The data has been downsampled by a factor of 2 in time and frequency to make the pulse stand out.
• Figure 2: Demonstration of the chosen RFI removal algorithms, run individually, on the data shown in Fig. 1. The red lines at the left of the top two panels indicate the channels that are flagged as RFI-affected by the respective algorithms. filtool (used for SKF and ZDMF) corrects for the bandshape of the chunk of data, whereas iqrm-apollo does not. To present the data comparable to each other, the topmost plot in the figure showing data cleaned by IQRM is, therefore also shown after correcting for its bandshape. Since ZDMF acts on the time samples, it does not mask any data but changes every time sample, and so only removes broadband RFI. One can also notice the patterns extending the residual RFI across all the frequency channels.
• Figure 3: Histogram of the number of true and false positive candidates detected with and without RFIM in a single pulse search. A test vector containing 6 pulses with a DM of $300\,\mathrm{pc/cm^3}$, a width of $\mathrm{40\,ms}$, a single-pulse S/N of $42.4$, and a combination of all three types of RFI ($2\sigma$) is searched for single pulses. Once without RFI cleaning and once after cleaning using IQRM and ZDMF.
• Figure 4: Detection rates for different combinations of RFIM applied to test vectors containing realistic RFI as a function of DM are shown. The fractions of recovered pulses ($\mathrm{F_p}$) are plotted against $\mathrm{DM}$ for multiple widths. The combinations of RFIM used are - IQRM, IQRM + ZDMF and SKF + ZDMF. The plots on the left side are obtained from results when test vectors from TVS-2 containing RFI of type 8 are cleaned with respective RFIM, and the ones on the right side contain RFI of type 1 (see Table \ref{['tab:realistic_rfi_testvectors']}). The shaded region indicates the range of $\mathrm{F_p}$ for different S/N (ranging from 14.1, corresponding to the lower edge, to 141.4, corresponding to the upper edge) at a given DM for a given pulse width. In the first panel, where the test vectors containing RFI of type 8 are cleaned using IQRM, no pulses are recovered, but the fraction of recovery increases when the RFI is weaker and/or cleaned by ZDMF as well.
• Figure 5: The impact of each algorithm on the fraction of recovered pulses ($\mathrm{F_p}$) across a range of DMs for test vectors without added RFI. The test vectors contain pulses with a width of $\mathrm{40,ms}$ and a (S/N) of 42.4. Note that the $\mathrm{F_p}$ curves for SKF and IQRM overlap in the figure.