Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study

TL;DR

This study introduces continual learning for engineering regression tasks by defining three regression-specific CL scenarios and benchmarking three established strategies (Experience Replay, Elastic Weight Consolidation, Gradient Episodic Memory) across five 3D engineering datasets. It demonstrates that replay-based methods, particularly Experience Replay, can substantially mitigate forgetting while delivering competitive final accuracy and notable reductions in training time compared with joint retraining. The framework leverages two surrogate-model architectures (Regression PointNet for point clouds and Regression ResNet for parametric data) and uses regression-specific metrics (MPE, MAE) and forgetting measures to evaluate performance across sequential experiences. The work provides practical insights into the memory-time-accuracy trade-offs and emphasizes the relevance of data representation and task sequencing for real-world engineering design under evolving data regimes. By delivering nine engineered CL benchmarks, the paper offers a transferable evaluation suite to promote the adoption of continual learning in surrogate modeling for dynamic engineering design pipelines.

Abstract

Engineering problems that apply machine learning often involve computationally intensive methods but rely on limited datasets. As engineering data evolves with new designs and constraints, models must incorporate new knowledge over time. However, high computational costs make retraining models from scratch infeasible. Continual learning (CL) offers a promising solution by enabling models to learn from sequential data while mitigating catastrophic forgetting, where a model forgets previously learned mappings. This work introduces CL to engineering design by benchmarking several CL methods on representative regression tasks. We apply these strategies to five engineering datasets and construct nine new engineering CL benchmarks to evaluate their ability to address forgetting and improve generalization. Preliminary results show that applying existing CL methods to these tasks improves performance over naive baselines. In particular, the Replay strategy achieved performance comparable to retraining in several benchmarks while reducing training time by nearly half, demonstrating its potential for real-world engineering workflows. The code and datasets used in this work will be available at: https://github.com/kmsamuel/cl-for-engineering-release.

Figures (7)

• Figure 1: Catastrophic forgetting is demonstrated using the car drag prediction problem from the DrivAerNet dataset. A surrogate model is trained on each data batch as the batches incrementally become available. Fig. \ref{['fig:forgetting_kdes']} shows a KDE plot of the final model's predictions on the entire dataset, which consists of all batches. As seen, the model exclusively predicts values within the third and fourth batches, demonstrating a lack of retention of earlier batch samples. Fig \ref{['fig:forgetting_errors']} emphasizes this discrepancy by reporting the percent error in predictions for each batch.
• Figure 2: Continual learning model training scheme, in which a model is incrementally trained on new experiences and evaluated on all experiences seen up until that point.
• Figure 3: Proposed regression continual learning scenarios that would be relevant for engineering. Figure \ref{['fig:bin_inc']} depicts the bin incremental scenario, in which the data is divided into experiences based on binning the target values. Figure \ref{['fig:input_inc']} depicts the input incremental scenario, in which the data is divided into experiences based on input categorizations, shown as clusters of the input data in the figure. Figure \ref{['fig:multi_targ']} depicts the multi-target incremental scenario, in which the data is divided into experiences based on newly desired targets, meaning each experience contains a full dataset, but a new predictive target.
• Figure 4: Continual learning benchmarks are defined as the combination of a dataset with a continual learning scenario.
• Figure 5: Overview of the continual learning strategies evaluated in this work. In Figure \ref{['fig:replay_fig']}, Experience Replay (ER) reintroduces stored samples from previous experiences during training via a memory buffer. In Figure \ref{['fig:ewc_fig']}, Elastic Weight Consolidation (EWC) adds a regularization term based on the Fisher Information Matrix to preserve important parameters from earlier tasks. In Figure \ref{['fig:gem_fig']}, Gradient Episodic Memory (GEM) constrains the loss function using stored samples from previous experiences, based on a specified number of patterns per experience (PPE). The * symbol indicates where past knowledge is integrated into the training process for each method.