SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling

TL;DR

SegNet introduces a fully trainable encoder-decoder CNN that learns to upsample deep, low-resolution feature maps via pooled-index guided decoding to produce pixel-wise semantic labels. The flat architecture maintains constant feature depth, uses 7×7 kernels for broad contextual awareness, and trains in a modular layer-wise fashion, enabling deeper networks without excessive parameter growth. Empirical results on CamVid, KITTI, and NYU-v2 show CRF-free, smooth, and competitive segmentation performance, with clear qualitative benefits in labeling small or thin structures. The work also demonstrates informative visualization of feature activations and transferability across datasets, highlighting SegNet’s practical impact for robust scene understanding.

Abstract

We propose a novel deep architecture, SegNet, for semantic pixel wise image labelling. SegNet has several attractive properties; (i) it only requires forward evaluation of a fully learnt function to obtain smooth label predictions, (ii) with increasing depth, a larger context is considered for pixel labelling which improves accuracy, and (iii) it is easy to visualise the effect of feature activation(s) in the pixel label space at any depth. SegNet is composed of a stack of encoders followed by a corresponding decoder stack which feeds into a soft-max classification layer. The decoders help map low resolution feature maps at the output of the encoder stack to full input image size feature maps. This addresses an important drawback of recent deep learning approaches which have adopted networks designed for object categorization for pixel wise labelling. These methods lack a mechanism to map deep layer feature maps to input dimensions. They resort to ad hoc methods to upsample features, e.g. by replication. This results in noisy predictions and also restricts the number of pooling layers in order to avoid too much upsampling and thus reduces spatial context. SegNet overcomes these problems by learning to map encoder outputs to image pixel labels. We test the performance of SegNet on outdoor RGB scenes from CamVid, KITTI and indoor scenes from the NYU dataset. Our results show that SegNet achieves state-of-the-art performance even without use of additional cues such as depth, video frames or post-processing with CRF models.

Figures (6)

• Figure 1: A 4 layer SegNet which takes in an RGB input image and performs feed-forward computation to obtain pixel-wise labelling. A stack of feature encoders is followed by a corresponding decoders. The soft-max layer classifies each pixel independently using the features input by the last decoder. An encoder uses the convolution-ReLU-max pooling-subsampling pipeline. A decoder upsamples its input using the transferred pool indices from its encoder. It then performs convolution with a trainable filter bank.
• Figure 2: (a) The modular training starts by optimizing the first encoder and decoder weights. The soft-max can be pre-trained or randomly initialized. (b) Once the first pair is trained, we insert an inner deeper encoder-decoder pair and optimize these weights while holding the outer encoder-decoder and soft-max weights fixed. More deeper pairs are then subsequently trained. Note encoder and decoder weights are untied.
• Figure 3: SegNet feature ablation study. All layers use 64 features in the SegNet. The four columns from right to left show the predictions obtained at various depths when only a fraction of the feature activations are used and the remaining set to zero. Note the quality of the labelling improves strikingly at depth 4 even when only the top-1 feature activations are considered. Interestingly, these activations seem to be tuned largely for the static scene classes and the other classes such as cars are labelled only when more features are activated. When fewer features are activated, missing categories (cars) are filled with sidewalk which is reasonable. Also shown are the percentage of activated features as part of the top 'N' activations; deeper layers have fewer but more finely tuned activations.
• Figure 4: Result samples on CamVid day and dusk test sequences. The evolution of various unary predictions and unaries combined with externally trained detectors LadickyECCV and CRF models Sturgess. SegNet predictions retain small categories such as poles (column 2,4), bicyclist (column 3), far side sidewalk (column 2) better than other methods while producing overall smooth predictions. CRF results, although smooth, miss several important categories even when SfM based cues are used. In the dusk scenario, SfM cues are particularly valuable (row 3). Here the SegNet fails to label the car (column 4) however, it fills this part with very reasonable predictions.
• Figure 5: Row 1,2 show KITTI test samples. Notice the illumination differences to CamVid samples in Fig. \ref{['CamVidQualy']}. Row 3: Predictions when all layers are trained afresh from a random initialization using the KITTI training set. Row 4: SegNet is pre-trained with the CamVid dataset and only layer 4 is trained for two epochs on the KITTI training set. Supervised pre-training can produce good results with a small extra computational effort. Row 5: poor results obtained when starting from pre-trained weights and training only a soft-max classifier with a hidden layer. Unknown class is blackened.