Training from Zero: Radio Frequency Machine Learning Data Quantity Forecasting

TL;DR

The paper tackles estimating the amount of training data needed to achieve a target performance in RFML AMC using minimal seed data. It leverages transfer-learning metrics including $NCE$, $LEEP$, and $LogME$ to predict data needs by regressing their scores against data quantity per class ($OPC$) via a log-linear model, and introduces a label whitening scheme controlled by $\epsilon$ to map metrics to performance targets. Across data origins $\Omega_C$, $\Omega_S$, and $\Omega_A$ and various waveform sets, $NCE$ and $LEEP$ provide robust quantity predictions while $LogME$ offers strong bounds for augmented data; a midpoint between $NCE$ and $LogME$ balances under- and overestimation for practical planning. The framework enables data-collection planning with limited seeds and can generalize to other classification tasks by quantifying dataset quality via transfer-learning metrics.

Abstract

The data used during training in any given application space is directly tied to the performance of the system once deployed. While there are many other factors that go into producing high performance models within machine learning, there is no doubt that the data used to train a system provides the foundation from which to build. One of the underlying rule of thumb heuristics used within the machine learning space is that more data leads to better models, but there is no easy answer for the question, "How much data is needed?" This work examines a modulation classification problem in the Radio Frequency domain space, attempting to answer the question of how much training data is required to achieve a desired level of performance, but the procedure readily applies to classification problems across modalities. The ultimate goal is determining an approach that requires the least amount of data collection to better inform a more thorough collection effort to achieve the desired performance metric. While this approach will require an initial dataset that is germane to the problem space to act as a \textit{target} dataset on which metrics are extracted, the goal is to allow for the initial data to be orders of magnitude smaller than what is required for delivering a system that achieves the desired performance. An additional benefit of the techniques presented here is that the quality of different datasets can be numerically evaluated and tied together with the quantity of data, and ultimately, the performance of the architecture in the problem domain.

Figures (7)

• Figure S1: (a) A visualization of how the generalized problem space, $\mathcal{S}$, encompasses the application space, $\mathcal{V}$, as well as all possible data collection methods, $\mathcal{T}$. (b) The process of sampling from a collection method, $\mathcal{T}$, in order to produce a training dataset, $\mathcal{Y}$. (c) The sampling of data from the application space to produce an evaluation dataset, $\mathcal{X}$, for estimating a trained model's performance if used within the application space. (d) The training process, $g(\cdot)$, with a given architecture, $f(\cdot)$, and training set, $\mathcal{Y}$, to produce the parameters, $\theta$, that can be used for inference with the architecture, $f(\cdot;\theta)\equiv\phi(\cdot)$. (e) The inference process using a trained model on the evaluation dataset, $\breve{l}_{y|x}$.
• Figure S2: DL Neural Network Architecture for the Convolutional Long Short-Term Memory (LSTM) Deep Neural Network (CLDNN) Architecture used in this work as the DL approach for the 10-class waveform AMC problem space. The model consists of multiple layers of convolution layers each followed by non-linear activation and regularization that are combined along the channel dimension before passing through a recurrent LSTM layer and finally passing through linear layers followed by non-linear activation and regularization to produce the model's inference.
• Figure S3: Visualization of the relationships between the three metrics (Left column: NCE, Middle column: LEEP, Right column: LogME) and the performance metric (Accuracy) of each network when measured on the results of the evaluation set $\Omega_{TC}$, or the target dataset in TL vernacular. Each dataset used for training are positioned along the rows (Top row: $\Omega_{C}$, Middle row: $\Omega_{S}$, Bottom row: $\Omega_{A}$). Linear trends shown between the metrics and accuracy for better clarity in the relationships.
• Figure S4: Plots show the relationship between quantity of data used from each dataset (Top: $\Omega_{C}$, Middle: $\Omega_{S}$, Bottom: $\Omega_{A}$) and the Accuracy achieved by networks trained on that amount of data. In general, the networks trained from datasets $\Omega_{C}$ and $\Omega_{A}$ have an increasing relation, but network, trained using $\Omega_{S}$ have a stagnant relation to performance in regards to quantity of data used to train.
• Figure S5: Plots show the residuals between the regressed log-linear fits of quantity of data available during training and the accuracy of each trained network and the observed accuracy of each network. Plots show similar trends across the used datasets (Top: $\Omega_{C}$, Middle: $\Omega_{S}$, Bottom: $\Omega_{A}$), with the edges of the available data deviating in the same direction, indicating a log-linear fit is not the ideal relationship between data quantity and accuracy.