Conformalised data synthesis

TL;DR

This work addresses data-scarcity and quality concerns in deep learning by introducing conformalised data synthesis, which confines synthetic data generation to high-confidence regions in feature space using Mondrian Inductive Conformal Prediction. The method samples from label-conditional, confidence-regulated regions defined by a grid over the feature space and a KNN-based non-conformity measure, producing synthetic data that enhances model performance without relying on purely density-based assumptions. Empirical results across five benchmarks show substantial gains in F1-score, especially for small, imbalanced, or overlapping datasets, and even demonstrate successful synthetic replacement in USPS. While theoretical guarantees do not directly translate to synthesis performance, the approach provides a practical, statistically grounded framework for confidence-aware data generation and opens avenues for tighter integration with conformal methods and privacy-aware data augmentation.

Abstract

With the proliferation of increasingly complicated Deep Learning architectures, data synthesis is a highly promising technique to address the demand of data-hungry models. However, reliably assessing the quality of a 'synthesiser' model's output is an open research question with significant associated risks for high-stake domains. To address this challenge, we propose a unique synthesis algorithm that generates data from high-confidence feature space regions based on the Conformal Prediction framework. We support our proposed algorithm with a comprehensive exploration of the core parameter's influence, an in-depth discussion of practical advice, and an extensive empirical evaluation of five benchmark datasets. To show our approach's versatility on ubiquitous real-world challenges, the datasets were carefully selected for their variety of difficult characteristics: low sample count, class imbalance, and non-separability. In all trials, training sets extended with our confident synthesised data performed at least as well as the original set and frequently significantly improved Deep Learning performance by up to 61 percentage points F1-score.

Figures (20)

• Figure 1: Visualisation of two techniques to identify feature space regions from which data is synthesised (\ref{['fig:introConfidence', 'fig:introDensity']}). Compared to traditional density estimation of the samples, our proposed algorithm determines conformal high-confidence regions based on a user-selected threshold. The lower the threshold, the wider the confidence regions become from which samples are synthesised.
• Figure 2: Intuition of the $\epsilon$ trade-off for traditional conformal classification and conformal synthesis. A small $\epsilon$ implies low error rates while increasing the inclusion of false labels or 'unrepresentative' samples, respectively. Note that the latter and the graphs in (b) must be inferred from downstream model performances due to the data generation domain. The elbow method is a straightforward heuristic to select a value for $\epsilon$, balancing the opposing desires.
• Figure 3: The Deep Learning model architecture used in all trials. Design decisions were made to improve the versatility on different real-world datasets and the robustness of the results. The input layer size $d$ and output layer size $n$ are driven by the dataset's dimensionality and number of classes.
• Figure 4: The original, synthesised, and extended training sets evaluated with Deep Learning on the same held-out test set. Initially, Train$_{\text{orig}}$ is temporarily divided into the proper training (60%) and calibration sets (40%) for conformal synthesis.
• Figure 5: A straightforward 2-dimensional toy dataset used to illustrate the influence of the proposed algorithm's parameters.