A Multi-objective Optimization Benchmark Test Suite for Real-time Semantic Segmentation

TL;DR

The paper addresses the challenge of HW-NAS for real-time semantic segmentation by formulating it as a standard multi-objective optimization problem (MOP) and delivering CitySeg/MOP, a fifteen-instance benchmark integrated in EvoXBench. It introduces a 32-bit fixed-length architecture encoding, a surrogate model for the prediction error objective $f^e$, and look-up tables for complexity and hardware objectives, evaluated across two hardware platforms to capture real-time constraints. The contributions include the CitySeg/MOP formulation, a MoSegNAS-inspired search space, and comprehensive MOEA benchmarking demonstrating prediction accuracy, sample diversity, and evaluation efficiency, with practical demonstrations on MOEAs like NSGA-II/NSGA-III and IBEA/HypE. Overall, CitySeg/MOP enables standardized, hardware-aware evaluation of MOEAs for real-time segmentation, bridging research and deployment through EvoXBench's language-agnostic interfaces.

Abstract

As one of the emerging challenges in Automated Machine Learning, the Hardware-aware Neural Architecture Search (HW-NAS) tasks can be treated as black-box multi-objective optimization problems (MOPs). An important application of HW-NAS is real-time semantic segmentation, which plays a pivotal role in autonomous driving scenarios. The HW-NAS for real-time semantic segmentation inherently needs to balance multiple optimization objectives, including model accuracy, inference speed, and hardware-specific considerations. Despite its importance, benchmarks have yet to be developed to frame such a challenging task as multi-objective optimization. To bridge the gap, we introduce a tailored streamline to transform the task of HW-NAS for real-time semantic segmentation into standard MOPs. Building upon the streamline, we present a benchmark test suite, CitySeg/MOP, comprising fifteen MOPs derived from the Cityscapes dataset. The CitySeg/MOP test suite is integrated into the EvoXBench platform to provide seamless interfaces with various programming languages (e.g., Python and MATLAB) for instant fitness evaluations. We comprehensively assessed the CitySeg/MOP test suite on various multi-objective evolutionary algorithms, showcasing its versatility and practicality. Source codes are available at https://github.com/EMI-Group/evoxbench.

Figures (7)

• Figure 1: Architecture encoding. The search space is encoded as a 32-bit fixed-length integer-valued string.
• Figure 2: An example process of performance evaluation via fitness evaluator. The upper half and the bottom half indicate an example process of the look-up predictor and the surrogate model predictor respectively.
• Figure 3: Correlations between the measured and the predicted metrics. The Mean Absolute Error (MAE) and Pearson correlation coefficient ($\rho$) are shown in the corresponding subfigures.
• Figure 4: Distribution of $f^c_1$ and $f^c_2$ under randomly sampled architectures.
• Figure 5: Correlations between the measured and the predicted metrics. The measured and the predicted metrics are obtained from measurements and the benchmark test suite correspondingly.