VVTRec: Radio Interferometric Reconstruction through Visual and Textual Modality Enrichment

TL;DR

This work tackles the ill-posed problem of reconstructing clean sky images from sparse visibility data in radio interferometry by introducing VVTRec, a multimodal reconstruction framework. It converts sparse visibilities into image-form $\mathcal{M}^{v \rightarrow i}$ and text-form $\mathcal{M}^{v \rightarrow t}$ modalities and leverages Vision-Language Models to extract additional spatial and semantic cues, augmented by a sample-specific knowledge bank that fuses external pre-trained knowledge via cross-modal attention. A conditional reconstructor integrates these multimodal features, enabling training-free gains with limited overhead and strong performance across multiple datasets and VLM backbones. Empirical results show significant PSNR/SSIM improvements over unimodal baselines and favorable data efficiency, underscoring the practicality of multimodal information fusion for radio astronomy and suggesting extensions to other imaging domains like MRI and seismic imaging.

Abstract

Radio astronomy is an indispensable discipline for observing distant celestial objects. Measurements of wave signals from radio telescopes, called visibility, need to be transformed into images for astronomical observations. These dirty images blend information from real sources and artifacts. Therefore, astronomers usually perform reconstruction before imaging to obtain cleaner images. Existing methods consider only a single modality of sparse visibility data, resulting in images with remaining artifacts and insufficient modeling of correlation. To enhance the extraction of visibility information and emphasize output quality in the image domain, we propose VVTRec, a multimodal radio interferometric data reconstruction method with visibility-guided visual and textual modality enrichment. In our VVTRec, sparse visibility is transformed into image-form and text-form features to obtain enhancements in terms of spatial and semantic information, improving the structural integrity and accuracy of images. Also, we leverage Vision-Language Models (VLMs) to achieve additional training-free performance improvements. VVTRec enables sparse visibility, as a foreign modality unseen by VLMs, to accurately extract pre-trained knowledge as a supplement. Our experiments demonstrate that VVTRec effectively enhances imaging results by exploiting multimodal information without introducing excessive computational overhead.

Figures (7)

• Figure 1: Comparison of traditional unimodal visibility data processing paradigm and our multimodal pipeline.
• Figure 2: Overview of our proposed method VVTRec. It consists of four key components: (1) The transformations provide essential modality refinement and expansion for viewing visibility data from a multimodal perspective. (2) The generators encode rich and consistent information from three modalities into latent spaces for further alignment and extraction. (3) The knowledge bank retrieves the enriched multimodal features in a unified space by modelling the joint distribution of all modalities. (4) The conditional reconstructor is designed to combine extracted features into the reconstruction and imaging process.
• Figure 3: (a) Comparison of the generated image-form feature map and PCs of an image in the pretraining dataset lin2014microsoft. (b) An example of retrieving in-context information from sparse visibility.
• Figure 4: Illustration of the workflow of the sample-specific knowledge bank, along with the knowledge pass and integration.
• Figure 5: Visualization of the reconstruction results of VVTRec compared with the dirty image and real sky, including both visibility and image. The left half of the panel below is the real part, and the right half is the imaginary part.