Multiform Evolution for High-Dimensional Problems with Low Effective Dimensionality

TL;DR

An instantiation of the multiform optimization paradigm is presented, where multiple low-dimensional counterparts of a target high-dimensional task are generated via random embeddings, enabling the target task to efficiently reuse solutions evolved across various low-dimensional searches via cross-form genetic transfers, hence speeding up overall convergence characteristics.

Abstract

In this paper, we scale evolutionary algorithms to high-dimensional optimization problems that deceptively possess a low effective dimensionality (certain dimensions do not significantly affect the objective function). To this end, an instantiation of the multiform optimization paradigm is presented, where multiple low-dimensional counterparts of a target high-dimensional task are generated via random embeddings. Since the exact relationship between the auxiliary (low-dimensional) tasks and the target is a priori unknown, a multiform evolutionary algorithm is developed for unifying all formulations into a single multi-task setting. The resultant joint optimization enables the target task to efficiently reuse solutions evolved across various low-dimensional searches via cross-form genetic transfers, hence speeding up overall convergence characteristics. To validate the overall efficacy of our proposed algorithmic framework, comprehensive experimental studies are carried out on well-known continuous benchmark functions as well as a set of practical problems in the hyper-parameter tuning of machine learning models and deep learning models in classification tasks and Predator-Prey games, respectively.

Figures (6)

• Figure 1: Illustration of random embedding from $D=2$ to $d=1$. The box denotes the original constrained solution space $\mathcal{X}$, while the red line illustrates a compressed and bounded low-dimensional search space $\mathcal{Y}$.
• Figure 2: Multiform optimization combines multiple formulations of a target optimization task into one all-encompassing algorithm and optimizes all of the formulations in a concurrent manner.
• Figure 3: Illustrated functions (i.e., Ackley and Rastrigin) in $D=2$ dimensions only have $d=1$ effective dimension. According to random embedding matrices, $f(\textbf{x})$ is formulated into $d$-dimensional functions $g_1(\textbf{y})$ and $g_2(\textbf{y})$, which possess different forms of searching landscapes.
• Figure 4: Illustration of proposed multiform EA with random embeddings from $D=2$ to $d=1$. Note in this example, the original high-dimensional optimization problem $f(\mathbf{x})$ is formulated into two low-dimensional tasks $g_1(\mathbf{y})$ and $g_2(\mathbf{y})$. The multiform EA enables the cross-form genetic transfer across tasks $g_1(\mathbf{y})$ and $g_2(\mathbf{y})$.
• Figure 5: Objective values obtained by DECC-G, LM-CMA, CSO, OpenAI-ES, CMA-ES-ERDG and proposed multiform EAs on solving optimization functions averaged by 20 independent runs.