Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks

TL;DR

The paper addresses transferring non-differentiable surrogate models under unknown affine domain shifts by mapping target inputs via x' = W x + v, with $W \,in\, SO(d)$ and $v \,in\, \mathbb{R}^d$, and learning hat f^T(x) = hat f^S(x') using a small transfer set. The method optimizes the affine parameters with CMA-ES, parameterizing W through the Lie algebra so(d) via W = Exp(A). Empirical results on BBOB and four real-world benchmarks show data-efficient gains in low-data regimes, particularly in higher dimensions, but highlight limitations when the source fit is poor or domain shifts are nonlinear or large. The work opens avenues for extending affine-transfer to other non-differentiable surrogates and for integrating active learning to further improve sample efficiency in engineering and space applications.

Abstract

Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. However, constructing a high-quality surrogate model often demands extensive data acquisition. A solution to this issue is to transfer pre-trained surrogate models for new tasks, provided that certain invariances exist between tasks. This study focuses on transferring non-differentiable surrogate models (e.g., random forests) from a source function to a target function, where we assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target. Previous research attempts to tackle this challenge for differentiable models, e.g., Gaussian process regression, which minimizes the empirical loss on the transfer data by tuning the affine transformations. In this paper, we extend the previous work to the random forest and assess its effectiveness on a widely-used artificial problem set - Black-Box Optimization Benchmark (BBOB) testbed, and on four real-world transfer learning problems. The results highlight the significant practical advantages of the proposed method, particularly in reducing both the data requirements and computational costs of training surrogate models for complex real-world scenarios.

Figures (22)

• Figure 1: Synthetic transfer learning tasks are constructed by transferring from different instances of the same BBOB functions (F3 Rastrigin is taken in this example). From left to right, we show the source function$f^{\text{S}}$, the target function$f^{\text{T}}$, the original RFR$\hat{f}^{\text{S}}$ trained to approximate $f^{\text{S}}$, the RFR trained directly on 50 samples of $f^{\text{T}}$, and the original RFR transferred with the same 50 samples. We also show the RFR models with 100 sample points.
• Figure 2: Transfer Learning via Learning Affine Transformation with CMA-ES
• Figure 3: The comparison evaluates Random forest regression models obtained through transfer learning against those trained from scratch on the transfer dataset, across varying sample sizes and problem dimensions. Each cell in the figure shows the percentage difference in average SMAPE (%) between the two approaches for specific BBOB functions. Positive values (marked in red) indicate better accuracy from transfer learning (i.e., lower SMAPE).
• Figure 4: The comparison evaluates Random forest regression models obtained through transfer learning against those trained from scratch on Porkchop Plot Benchmarks for Earth-to-Mars trajectory optimization. It examines different transfer data sample sizes, with each figure cell showing the percentage difference in average SMAPE (%) between the two approaches for specific transfer settings. Positive values indicate better accuracy from transfer learning (i.e. lower SMAPE).
• Figure 5: The comparison evaluates Random forest regression models obtained through transfer learning against those trained from scratch on the Kinematics of a Robot Arm. It examines different transfer data sample sizes, with each figure cell showing the percentage difference in average SMAPE (%) between the two approaches for specific transfer settings. Positive values indicate better accuracy from transfer learning (i.e. lower SMAPE).