FBNetV5: Neural Architecture Search for Multiple Tasks in One Run

TL;DR

FBNetV5 introduces a multitask neural architecture search framework that designs backbones for image classification, object detection, and semantic segmentation in a single run. It combines a simple, transferable block-wise search space with a disentangled multitask training process and a constant-cost search algorithm that uses importance sampling and REINFORCE gradients to keep compute independent of task count. Empirical results show state-of-the-art performance across ImageNet, ADE20K, and COCO with reduced FLOPs and training effort, demonstrating strong cross-task transferability. The approach significantly lowers human effort and computational demands for multitask NAS, enabling scalable backbone design across diverse CV tasks.

Abstract

Neural Architecture Search (NAS) has been widely adopted to design accurate and efficient image classification models. However, applying NAS to a new computer vision task still requires a huge amount of effort. This is because 1) previous NAS research has been over-prioritized on image classification while largely ignoring other tasks; 2) many NAS works focus on optimizing task-specific components that cannot be favorably transferred to other tasks; and 3) existing NAS methods are typically designed to be "proxyless" and require significant effort to be integrated with each new task's training pipelines. To tackle these challenges, we propose FBNetV5, a NAS framework that can search for neural architectures for a variety of vision tasks with much reduced computational cost and human effort. Specifically, we design 1) a search space that is simple yet inclusive and transferable; 2) a multitask search process that is disentangled with target tasks' training pipeline; and 3) an algorithm to simultaneously search for architectures for multiple tasks with a computational cost agnostic to the number of tasks. We evaluate the proposed FBNetV5 targeting three fundamental vision tasks -- image classification, object detection, and semantic segmentation. Models searched by FBNetV5 in a single run of search have outperformed the previous stateof-the-art in all the three tasks: image classification (e.g., +1.3% ImageNet top-1 accuracy under the same FLOPs as compared to FBNetV3), semantic segmentation (e.g., +1.8% higher ADE20K val. mIoU than SegFormer with 3.6x fewer FLOPs), and object detection (e.g., +1.1% COCO val. mAP with 1.2x fewer FLOPs as compared to YOLOX).

Figures (5)

• Figure 1: The architectures simultaneously searched in a single run of FBNetV5 outperforms the SotA performance in three tasks: ImageNet deng2009imagenet image classification, ADE20K zhou2017scene semantic segmentation, and COCO lin2014microsoft object detection.
• Figure 2: Overview of FBNetV5. We search backbone topologies for multiple tasks by training a supernet once on a multitask dataset. Each task has its own architecture distribution from which we sample task-specific architectures and train them using the existing training pipeline of the target tasks. Supernet configurations in Appendix \ref{['sec:block_config']}. Fusion module details in Appendix \ref{['sec:fusion_details']}. Search process in Algorithm \ref{['alg:multitask_supernet_is_reinforce']}.
• Figure 3: Visualization of the searched architectures for semantic segmentation (SEG), object detection (DET), and image classification (CLS) tasks.
• Figure 4: Illustration of the fusion module aggregating information from different paths (resolutions).
• Figure 5: The search space can represent the topology of U-Net ronneberger2015u, PANet liu2018path, FPN lin2017feature, and BiFPN (without the extra edge) tan2020efficientdet.