Can Synthetic Data Improve Symbolic Regression Extrapolation Performance?

TL;DR

This work investigates whether synthetic data generated in KDE-identified extrapolation regions via a knowledge-distillation framework can improve extrapolation in symbolic regression using GP. A teacher-student setup (M1 generates synthetic data, M2 learns from augmented data) is evaluated across six datasets with various teacher models (NN, RF, GPp, GPe) and GP as the student, focusing on extrapolation and heterogeneous error patterns. Findings show dataset-dependent benefits, with particularly reliable gains when GPe-generated data trains GPp, and a practical strategy to select augmentation using interpolation-region validation to avoid degradation. The proposed KDE-guided synthetic data augmentation offers a pragmatic approach to enhance extrapolation in SR, though results vary by dataset and model pairing.

Abstract

Many machine learning models perform well when making predictions within the training data range, but often struggle when required to extrapolate beyond it. Symbolic regression (SR) using genetic programming (GP) can generate flexible models but is prone to unreliable behaviour in extrapolation. This paper investigates whether adding synthetic data can help improve performance in such cases. We apply Kernel Density Estimation (KDE) to identify regions in the input space where the training data is sparse. Synthetic data is then generated in those regions using a knowledge distillation approach: a teacher model generates predictions on new input points, which are then used to train a student model. We evaluate this method across six benchmark datasets, using neural networks (NN), random forests (RF), and GP both as teacher models (to generate synthetic data) and as student models (trained on the augmented data). Results show that GP models can often improve when trained on synthetic data, especially in extrapolation areas. However, the improvement depends on the dataset and teacher model used. The most important improvements are observed when synthetic data from GPe is used to train GPp in extrapolation regions. Changes in interpolation areas show only slight changes. We also observe heterogeneous errors, where model performance varies across different regions of the input space. Overall, this approach offers a practical solution for better extrapolation. Note: An earlier version of this work appeared in the GECCO 2025 Workshop on Symbolic Regression. This arXiv version corrects several parts of the original submission.

Figures (5)

• Figure 1: Knowledge Distillation Framework for Synthetic Data Augmentation. First, a teacher model (M1) is trained on original data and used to generate synthetic extrapolation data. This synthetic data is added to the original training set. Then, a student model (M2) is trained twice: once on the original data, and once on the augmented dataset. The performance difference between both M2 runs is measured to evaluate the benefit of synthetic extrapolation data.
• Figure 2: Significance heatmaps ($1 = p< 0.05$) for interpolation (left) and extrapolation (right) across all datasets. Each cell indicates whether the performance difference between the baseline model (M2 Alone) and the augmented model (M2 after M1) is statistically significant. A value of 1 (dark colour) means the performance difference is statistically significant at $p < 0.05$, and 0 (light colour) means the difference is not statistically significant.
• Figure 3: Data distribution in the Gurson dataset (Run 1), plotted for $X_1$ and $X_2$ axes. The plot shows the locations of the training data, test interpolation points, test extrapolation points, and synthetic data. Extrapolation points lie far outside the training area, while synthetic data aims to help models generalise better in these outers areas.
• Figure 4: Absolute residuals of a GP model (Run 1) on the Gurson dataset, plotted against KDE density values. This plot shows how prediction error changes depending on how dense the data is. A slight upward trend in test extrapolation points (green) shows that error increases as data becomes more sparse.
• Figure 5: Absolute residuals of a GP model (Run 1) on the Gurson dataset, plotted against distance from centroid of the training data. This shows that prediction errors tend to grow the further the input is from the training data, especially in extrapolation areas.