High-Resolution Representations for Labeling Pixels and Regions

TL;DR

The paper addresses the need for precise, high-resolution representations across vision tasks. By aggregating upsampled features from all parallel resolutions, HRNetV2 builds stronger high-resolution representations and extends them to multi-level detection heads. The approach achieves state-of-the-art results in semantic segmentation, facial landmark detection, and COCO object detection, with efficient computation. The work demonstrates that simple multi-resolution feature fusion can substantially boost performance across diverse pixel- and region-labeling tasks, with public code and pretrained models released.

Abstract

High-resolution representation learning plays an essential role in many vision problems, e.g., pose estimation and semantic segmentation. The high-resolution network (HRNet)~\cite{SunXLW19}, recently developed for human pose estimation, maintains high-resolution representations through the whole process by connecting high-to-low resolution convolutions in \emph{parallel} and produces strong high-resolution representations by repeatedly conducting fusions across parallel convolutions. In this paper, we conduct a further study on high-resolution representations by introducing a simple yet effective modification and apply it to a wide range of vision tasks. We augment the high-resolution representation by aggregating the (upsampled) representations from all the parallel convolutions rather than only the representation from the high-resolution convolution as done in~\cite{SunXLW19}. This simple modification leads to stronger representations, evidenced by superior results. We show top results in semantic segmentation on Cityscapes, LIP, and PASCAL Context, and facial landmark detection on AFLW, COFW, $300$W, and WFLW. In addition, we build a multi-level representation from the high-resolution representation and apply it to the Faster R-CNN object detection framework and the extended frameworks. The proposed approach achieves superior results to existing single-model networks on COCO object detection. The code and models have been publicly available at \url{https://github.com/HRNet}.

Figures (5)

• Figure 1: A simple example of a high-resolution network. There are four stages. The $1$st stage consists of high-resolution convolutions. The $2$nd ($3$rd, $4$th) stage repeats two-resolution (three-resolution, four-resolution) blocks. The detail is given in Section \ref{['sec:HRNetV1']}.
• Figure 2: Multi-resolution block: (a) multi-resolution group convolution and (b) multi-resolution convolution. (c) A normal convolution (left) is equivalent to fully-connected multi-branch convolutions (right).
• Figure 3: (a) The high-resolution representation proposed in SunXLW19 (HRNetV$1$ ); (b) Concatenating the (upsampled) representations that are from all the resolutions for semantic segmentation and facial landmark detection (HRNetV$2$ ); (c) A feature pyramid formed over (b) for object detection (HRNetV$2$p). The four-resolution representations at the bottom in each sub-figure are outputted from the network in Figure \ref{['fig:HRNet']}, and the gray box indicates how the output representation is obtained from the input four-resolution representations.
• Figure 4: Empirical analysis. (a) Segmentation on Cityscapes val and PASCAL-Context test for comparing HRNetV$1$ and its variant HRNetV$1$h, and HRNetV$2$ (single scale and no flipping). (b) Object detection on COCO val for comparing HRNetV$1$ and its variant HRNetV$1$h, and HRNetV$2$p (LS = learning schedule).
• Figure 5: Representation for ImageNet classification. The input of the box is the representations of four resolutions.