Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Generalization in Evolutionary Feature Construction for Symbolic Regression through Vicinal Jensen Gap Minimization

Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang

TL;DR

An evolutionary feature construction framework that jointly optimizes empirical risk and the vicinal Jensen gap to control overfitting is proposed and comparisons with 15 machine learning algorithms indicate that genetic programming with the proposed overfitting control strategy achieves superior performance.

Abstract

Genetic programming-based feature construction has achieved significant success in recent years as an automated machine learning technique to enhance learning performance. However, overfitting remains a challenge that limits its broader applicability. To improve generalization, we prove that vicinal risk, estimated through noise perturbation or mixup-based data augmentation, is bounded by the sum of empirical risk and a regularization term-either finite difference or the vicinal Jensen gap. Leveraging this decomposition, we propose an evolutionary feature construction framework that jointly optimizes empirical risk and the vicinal Jensen gap to control overfitting. Since datasets may vary in noise levels, we develop a noise estimation strategy to dynamically adjust regularization strength. Furthermore, to mitigate manifold intrusion-where data augmentation may generate unrealistic samples that fall outside the data manifold-we propose a manifold intrusion detection mechanism. Experimental results on 58 datasets demonstrate the effectiveness of Jensen gap minimization compared to other complexity measures. Comparisons with 15 machine learning algorithms further indicate that genetic programming with the proposed overfitting control strategy achieves superior performance.

Enhancing Generalization in Evolutionary Feature Construction for Symbolic Regression through Vicinal Jensen Gap Minimization

TL;DR

An evolutionary feature construction framework that jointly optimizes empirical risk and the vicinal Jensen gap to control overfitting is proposed and comparisons with 15 machine learning algorithms indicate that genetic programming with the proposed overfitting control strategy achieves superior performance.

Abstract

Genetic programming-based feature construction has achieved significant success in recent years as an automated machine learning technique to enhance learning performance. However, overfitting remains a challenge that limits its broader applicability. To improve generalization, we prove that vicinal risk, estimated through noise perturbation or mixup-based data augmentation, is bounded by the sum of empirical risk and a regularization term-either finite difference or the vicinal Jensen gap. Leveraging this decomposition, we propose an evolutionary feature construction framework that jointly optimizes empirical risk and the vicinal Jensen gap to control overfitting. Since datasets may vary in noise levels, we develop a noise estimation strategy to dynamically adjust regularization strength. Furthermore, to mitigate manifold intrusion-where data augmentation may generate unrealistic samples that fall outside the data manifold-we propose a manifold intrusion detection mechanism. Experimental results on 58 datasets demonstrate the effectiveness of Jensen gap minimization compared to other complexity measures. Comparisons with 15 machine learning algorithms further indicate that genetic programming with the proposed overfitting control strategy achieves superior performance.
Paper Structure (48 sections, 2 theorems, 12 equations, 24 figures, 12 tables, 2 algorithms)

This paper contains 48 sections, 2 theorems, 12 equations, 24 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vicinal Risk Minimization
  4. Vicinal Risk Minimization in Deep Learning
  5. Vicinal Risk Minimization in GP
  6. Overfitting Control for GP
  7. Evolutionary Feature Construction
  8. Vicinal Risk Decomposition
  9. The Proposed Algorithm
  10. Algorithm Framework
  11. Empirical Vicinal Risk Decomposition
  12. Noise Estimation Strategy
  13. Manifold Intrusion Detection
  14. Experimental Settings
  15. Datasets
  16. ...and 33 more sections

Key Result

Theorem 1

Let $f: \mathcal{X} \rightarrow \mathcal{Y}$ be a machine learning model mapping from the input space $\mathcal{X}$ to the output space $\mathcal{Y}$. Consider the VRM framework where each input $x_i \in \mathcal{X}$ is perturbed by a noise vector $\epsilon$, resulting in a vicinal training instance

Figures (24)

  • Figure 1: Illustrative examples of Gaussian noise-based and mixup-based VRM. The predictions of the original and synthesized samples made by the fluctuating function are shown.
  • Figure 2: Workflow of Vicinal Jensen Gap Minimization-based GP.
  • Figure 3: Estimating Noise Level with Extremely Randomized Trees.
  • Figure 4: An example of manifold intrusion, where the ground truth and the prediction by Extra Trees overlap, but the synthesized sample deviates significantly from these two.
  • Figure 5: Comparison between optimizing vicinal Jensen gap versus finite difference. ("+","$\sim$", and "-" represent the number of datasets where optimizing vicinal Jensen gap performs better, similar, or worse, respectively, compared to optimizing finite difference.)
  • ...and 19 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • proof
  • proof