HyCTAS: Multi-Objective Hybrid Convolution-Transformer Architecture Search for Real-Time Image Segmentation

TL;DR

HyCTAS tackles real-time semantic and panoptic segmentation by automatically discovering efficient, high-resolution networks that hybridize lightweight convolutions with memory-efficient self-attention. It introduces a multi-branch HRNet-like supernet and uses NSGA-II-based multi-objective search to produce an approximate Pareto front of architectures balancing mIoU and latency, without ImageNet pretraining. The method achieves strong results on Cityscapes, ADE20K, and COCO, delivering competitive accuracy with real-time or near-real-time performance and revealing patterns in where to place attention across branches. These findings offer a practical pathway to deploy high-resolution segmentation models that meet strict speed and memory budgets in real-world settings.

Abstract

Real-time image segmentation demands architectures that preserve fine spatial detail while capturing global context under tight latency and memory budgets. Image segmentation is one of the most fundamental problems in computer vision and has drawn a lot of attention due to its vast applications in image understanding and autonomous driving. However, designing effective and efficient segmentation neural architectures is a labor-intensive process that may require numerous trials by human experts. In this paper, we address the challenge of integrating multi-head self-attention into high-resolution representation CNNs efficiently by leveraging architecture search. Manually replacing convolution layers with multi-head self-attention is non-trivial due to the costly overhead in memory to maintain high resolution. By contrast, we develop a multi-target multi-branch supernet method, which not only fully utilizes the advantages of high-resolution features but also finds the proper location for placing the multi-head self-attention module. Our search algorithm is optimized towards multiple objectives (e.g., latency and mIoU) and is capable of finding architectures on the approximate Pareto front with an arbitrary number of branches in a single search. We further present a series of models via the Hybrid Convolutional-Transformer Architecture Search (HyCTAS) method that searches for the best hybrid combination of lightweight convolution layers and memory-efficient self-attention layers between branches from different resolutions and fuses them at high resolution for both efficiency and effectiveness. On Cityscapes, ADE20K, and COCO, HyCTAS discovers competitive real-time models without ImageNet pretraining, delivering strong accuracy and latency trade-offs. Code and models are available at https://github.com/MarvinYu1995/HyCTAS.

Figures (5)

• Figure 1: An illustration of our HyCTAS search space. It preserves the multi-resolution and multi-branch properties of HRNet by introducing two types of searchable components: cells and nodes, which extend HRNet with multi-module fusion ability. Blue circles denote cells (operator selection; lightweight conv or memory-efficient self-attention), yellow circles denote nodes (fusion/skip gates controlling inter-branch connections). Solid arrows indicate active connections for a sampled sub-network; dashed arrows indicate optional connections available in the supernet that may or may not be selected during search. Upsampled features are concatenated and fused by a $3\times3$ conv.
• Figure 2: An illustration of our searchable modules. We design a memory-efficient self-attention module and a lightweight convolution module and search for the best combination.
• Figure 3: The searched architectures under different objectives on the Cityscapes validation set. Our proposed HyCTAS approach can search for multiple efficient architectures with arbitrary branches in a single run. The color from light to dark indicates that our search results are getting better and better as the search progresses. The solid line represents architectures on the approximate Pareto front (non-dominated set). Best viewed in color.
• Figure 4: An illustration of our searched architectures with multiple branches. Green denotes lightweight convolution, red denotes memory-efficient self-attention, and gray indicates skip/fusion nodes.
• Figure 5: Visualization on COCO and Cityscapes validation set. The first row corresponds to COCO original images and the second row presents their semantic segmentation results. The third row corresponds to Cityscapes original images and the fourth row presents their semantic segmentation results. Best viewed in color. Zoom in for details; predictions retain fine boundaries due to high-resolution branches.