An Efficient Semantic Segmentation Decoder for In-Car or Distributed Applications

TL;DR

This work addresses the challenge of deploying transformer-based semantic segmentation (SegDeformer) in automotive settings under tight compute and bandwidth constraints. It introduces joint feature and task decoding (JD) for both in-car and distributed deployments, including a hyperprior-based efficient source codec and a transformer-based JD that operates on compressed bottleneck representations. The proposed approach delivers up to $11.7\times$ in-car fps improvements and state-of-the-art rate-distortion performance in distributed settings while drastically reducing cloud-side parameters. The results demonstrate that SegDeformer can be made practical for real-time vehicular use and scalable cloud-based perception with minimal computational burden on edge devices and the cloud alike.

Abstract

Modern automotive systems leverage deep neural networks (DNNs) for semantic segmentation and operate in two key application areas: (1) In-car, where the DNN solely operates in the vehicle without strict constraints on the data rate. (2) Distributed, where one DNN part operates in the vehicle and the other part typically on a large-scale cloud platform with a particular constraint on transmission bitrate efficiency. Typically, both applications share an image and source encoder, while each uses distinct (joint) source and task decoders. Prior work utilized convolutional neural networks for joint source and task decoding but did not investigate transformer-based alternatives such as SegDeformer, which offer superior performance at the cost of higher computational complexity. In this work, we propose joint feature and task decoding for SegDeformer, thereby enabling lower computational complexity in both in-car and distributed applications, despite SegDeformer's computational demands. This improves scalability in the cloud while reducing in-car computational complexity. For the in-car application, we increased the frames per second (fps) by up to a factor of $11.7$ ($1.4$ fps to $16.5$ fps) on Cityscapes and by up to a factor of $3.5$ ($43.3$ fps to $154.3$ fps) on ADE20K, while being on-par w.r.t.\ the mean intersection over union (mIoU) of the transformer-based baseline that doesn't compress by a source codec. For the distributed application, we achieve state-of-the-art (SOTA) over a wide range of bitrates on the mIoU metric, while using only $0.14$\% ($0.04$\%) of cloud DNN parameters used in previous SOTA, reported on ADE20K (Cityscapes).

Figures (9)

• Figure 1: High-level comparison of existing state-of-the-art vs. our proposed approaches in both in-car (green) and distributed (red) semantic segmentation applications. Here, $\mathbf{E}$ and $\mathbf{D}$ are the image encoder and task decoder, respectively. Blocks $\mathbf{FE}$ and $\mathbf{FD}$ represent the feature encoder and decoder, respectively, while $\mathbf{CE}$ and $\mathbf{CD}$ correspond to the compression encoder and decoder. $\mathbf{JD}$ represents the proposed transformer-based joint feature and task decoder.
• Figure 2: Network architecture of feature encoder$\mathbf{FE}$ and feature decoder$\mathbf{FD}$ utilized in Fig. \ref{['fig:high_level']}, taken from ahuja2023neural.
• Figure 3: Hyperprior architecture of compression encoder$\mathbf{CE}$ and compression decoder$\mathbf{CD}$ utilized in Fig. \ref{['fig:high_level']} for the distributed baseline, distributed JD baseline, and for the proposed transformer-based distributed JD, adopted from nazirjd.
• Figure 4: Detailed overview of the decoder architectures for in-car and distributed semantic segmentation applications. Fig. \ref{['fig:overview_architectures']}(a) shows baseline decoder$\mathbf{D}$, taken from segdeformer. Fig. \ref{['fig:overview_architectures']}(b) illustrates the proposed architecture for the joint feature and task decoder$\mathbf{JD}$.
• Figure 5: Context mining block utilized in Fig. \ref{['fig:overview_architectures']}, adapted from segdeformer.