Taylor-Sensus Network: Embracing Noise to Enlighten Uncertainty for Scientific Data

TL;DR

The Taylor-Sensus Network (TSNet) is proposed, which innovatively uses a Taylor series expansion to model complex, heteroscedastic noise and proposes a deep Taylor block for aware noise distribution and demonstrates superior performance over mainstream and state-of-the-art methods in experiments.

Abstract

Uncertainty estimation is crucial in scientific data for machine learning. Current uncertainty estimation methods mainly focus on the model's inherent uncertainty, while neglecting the explicit modeling of noise in the data. Furthermore, noise estimation methods typically rely on temporal or spatial dependencies, which can pose a significant challenge in structured scientific data where such dependencies among samples are often absent. To address these challenges in scientific research, we propose the Taylor-Sensus Network (TSNet). TSNet innovatively uses a Taylor series expansion to model complex, heteroscedastic noise and proposes a deep Taylor block for aware noise distribution. TSNet includes a noise-aware contrastive learning module and a data density perception module for aleatoric and epistemic uncertainty. Additionally, an uncertainty combination operator is used to integrate these uncertainties, and the network is trained using a novel heteroscedastic mean square error loss. TSNet demonstrates superior performance over mainstream and state-of-the-art methods in experiments, highlighting its potential in scientific research and noise resistance. It will be open-source to facilitate the community of "AI for Science".

Figures (10)

• Figure 1: In scientific ML, aleatoric and epistemic uncertainties can lead to issues such as overfitting and overconfidence. However, they also provide valuable insights for data exploration.
• Figure 2: The Taylor-Sensus Network (TSNet) framework. It integrates Noise-Aware Contrastive Learning Module (NCL), Data Density Perception Module (DPM), and Uncertainty Combination Operator (UCO). The Deep Taylor Block (DTB) in NCL is crucial for estimating the $(\mu_{n}, \Sigma_{n})$ parameters. And NCL introduces re-noising for feature $\textbf{X}$ to derive $(\mu_{CL}, \Sigma_{CL})$ via the shared-parameters DTB, which improves noise learning via $\mathcal{L}_{CL}$. DPM uses KL loss $\mathcal{L}_{KL}$ for density estimation $\mathcal{K}$ against KNN density mapping $\rho$, yielding density-sensitive weight $k_d$. UCO refines $(\mu_{n}, \Sigma_{n})$ to $(\mu, \Sigma)$ using $k_d$. The reparameterization trick generates $\widetilde{Y}$ under $\mathcal{N}(\mu, \Sigma)$ for heteroscedastic MSE loss $\mathcal{L}_{HMSE}$. $X_{aug} \sim \mathcal{N}(X, \sigma^2_{aug})$ represents data augmentation by sampling from the distribution $\mathcal{N}(X, \sigma^2_{aug})$.
• Figure 3: The Deep Taylor Block (DTB). According to heteroscedastic noise transformation (Equation (\ref{['eq:Taylor_res']}) and (\ref{['eq:gasdis']})), $F_{\mu}(\cdot)$, $F_{\Sigma_i}(\cdot)$, and $F_{\Sigma_o}(\cdot)$ predict the noise-free data label $\mu_{DTB}$, the feature noise distribution parameters $\Sigma_i(\textbf{X}_{DTB})$, and the system noise distribution parameters $\Sigma_o(\textbf{X}_{DTB})$, respectively.
• Figure 4: The Noise-Aware Contrastive Learning Module (NCL). Zero-mean noise $\epsilon$ following the distribution $\mathcal{N}(0, \Sigma_k(\mathbf{X}))$ is randomly introduced to the sample features $\mathbf{X}$. The noisy features $\mathbf{X} + \epsilon$ and the original sample $\mathbf{X}$ are both passed through the DTB to predict their respective means and variances. By minimizing the mean and maximizing the variance in this process, contrastive learning is achieved, thereby enhancing the perception of feature noise.
• Figure 5: Experimental results of the anti-noise experiments. Relative Performance Index (RPI) is calculated as $\text{MSE}_i / \text{MSE}_0$, quantifying models' normalized performance change under different noise ratios. (a) (g) shows the results on the different datasets.$\downarrow$indicates that lower evaluation results are preferable.