Challenges of Interaction in Optimizing Mixed Categorical-Continuous Variables

TL;DR

This work analyzes optimization of mixed binary-continuous variables, focusing on how interactions between discrete and continuous components hinder CatCMA. It introduces two strategies—warm-starting (WS) and hyper-representation (HR)—to address Type-I and Type-II interactions, respectively, culminating in the ICatCMA algorithm. Through synthetic experiments on problems with controllable interaction strength, the paper demonstrates that CatCMA struggles under strong interactions, while ICatCMA improves success rates when both interaction types are present, albeit with trade-offs in efficiency. The contributions offer a principled approach to handling interactions in mixed-variable optimization and highlight directions for balancing robustness and computational cost in real-world applications.

Abstract

Optimization of mixed categorical-continuous variables is prevalent in real-world applications of black-box optimization. Recently, CatCMA has been proposed as a method for optimizing such variables and has demonstrated success in hyper-parameter optimization problems. However, it encounters challenges when optimizing categorical variables in the presence of interaction between continuous and categorical variables in the objective function. In this paper, we focus on optimizing mixed binary-continuous variables as a special case and identify two types of variable interactions that make the problem particularly challenging for CatCMA. To address these difficulties, we propose two algorithmic components: a warm-starting strategy and a hyper-representation technique. We analyze their theoretical impact on test problems exhibiting these interaction properties. Empirical results demonstrate that the proposed components effectively address the identified challenges, and CatCMA enhanced with these components, named ICatCMA, outperforms the original CatCMA.

Figures (2)

• Figure 1: Success rates (over $100$ trials) of ICatCMA when $T_\mathrm{freeze}= A\cdot10^2\ell/\lambda$ with $A=[1, 2, 5, 10, 20]$ on $f_{III}$ with $\alpha=16$.
• Figure 2: The best-so-far $f$-value evaluated on $f_{II}$ with $\alpha=4$. Each optimizer runs on $20$ problem instances.