Self-Supervised Radio Pre-training: Toward Foundational Models for Spectrogram Learning

TL;DR

Adopting a Convolutional LSTM architecture for efficient spatio-temporal processing, Masked Spectrogram Modeling is introduced, a novel self-supervised learning approach for pretraining foundational DL models on radio signals.

Abstract

Foundational deep learning (DL) models are general models, trained on large, diverse, and unlabelled datasets, typically using self-supervised learning techniques have led to significant advancements especially in natural language processing. These pretrained models can be fine-tuned for related downstream tasks, offering faster development and reduced training costs, while often achieving improved performance. In this work, we introduce Masked Spectrogram Modeling, a novel self-supervised learning approach for pretraining foundational DL models on radio signals. Adopting a Convolutional LSTM architecture for efficient spatio-temporal processing, we pretrain the model with an unlabelled radio dataset collected from over-the-air measurements. Subsequently, the pretrained model is fine-tuned for two downstream tasks: spectrum forecasting and segmentation. Experimental results demonstrate that our methodology achieves competitive performance in both forecasting accuracy and segmentation, validating its effectiveness for developing foundational radio models.

Figures (12)

• Figure 1: A spectrogram and label pair for the segmentation task.
• Figure 2: A radio sentence created from the RRD dataset.
• Figure 3: Illustration of the proposed methodology for MSM pretraining and downstream task fine-tuning.
• Figure 4: Masked and original spectrogram pair.
• Figure 5: Illustration of a spectrogram and its corresponding resource grid for a block size of ($1$ ms, $5$ MHz).