DN-CL: Deep Symbolic Regression against Noise via Contrastive Learning

TL;DR

DN-CL employs two parameter-sharing encoders to embed data points from various data transformations into feature shields against noise, utilizing contrastive learning to distinguish between 'positive' noise-corrected pairs and 'negative' contrasting pairs.

Abstract

Noise ubiquitously exists in signals due to numerous factors including physical, electronic, and environmental effects. Traditional methods of symbolic regression, such as genetic programming or deep learning models, aim to find the most fitting expressions for these signals. However, these methods often overlook the noise present in real-world data, leading to reduced fitting accuracy. To tackle this issue, we propose \textit{\textbf{D}eep Symbolic Regression against \textbf{N}oise via \textbf{C}ontrastive \textbf{L}earning (DN-CL)}. DN-CL employs two parameter-sharing encoders to embed data points from various data transformations into feature shields against noise. This model treats noisy data and clean data as different views of the ground-truth mathematical expressions. Distances between these features are minimized, utilizing contrastive learning to distinguish between 'positive' noise-corrected pairs and 'negative' contrasting pairs. Our experiments indicate that DN-CL demonstrates superior performance in handling both noisy and clean data, presenting a promising method of symbolic regression.

Figures (6)

• Figure 1: The framework of DN-CL. The input $D$ is fed to two parameter-sharing encoders with two different transformations. $t_1$ denotes identity operation meaning no transformation is made. $t_2$ denotes adding noise into the input $D$ to get a different view of true mathematical expression. The hidden features $h_1$ and $h_2$ are the input of a projection layer obtaining vectors $z_1$ and $z_2$ to compute InfoNCE loss. Besides, we concat $h_1$ and $h_2$ as the data feature along with the expression label inputting to the decoder to predict the expression. The loss is composed of the InfoNCE and cross-entropy loss.
• Figure 2: The average $R^2$ of each methods in uni-variate benchmarks. The x-axis indicates descending noise level from 0.2 to 0.0. The y-axis represents the average $R^2$ across all benchmarks. The dash line, blue, orange, and red line represents ground truth, clean-model, clean-model-1, and our models, respectively.
• Figure 3: Heatmaps for the features outputted by the encoder between clean data and noise data (noise level $\eta=0.1$). The expression is Keijzer-15.
• Figure 4: The average $R^2$ of each methods for Livermore and Nguyen datasets. The $x$-axis indicates descending noise level from 0.1 to 0.0. The $y$-axis represents the average $R^2$ across all benchmarks.
• Figure 5: Results on AI Feynman datasets, $R^2$, simplified complexity, and inference time are displayed. Color/shape indicates level of noise added to the target variable.