Numerical Coordinate Regression with Convolutional Neural Networks

TL;DR

The paper tackles coordinate regression in images by introducing DSNT, a differentiable, parameter-free layer that converts spatial heatmaps into 2D coordinates via an expectation-based computation. DSNT preserves spatial generalization and end-to-end training, overcoming heatmap quantization and fully connected layer limitations. Empirical results on MPII pose estimation show DSNT, especially with regularization, outperforms heatmap matching and FC baselines across architectures and resolutions, while enabling faster and more memory-efficient inference. The work provides a practical, broadly applicable baseline for coordinate regression in fully convolutional networks and suggests future extensions to adversarial training and integration with Spatial Transformer Networks.

Abstract

We study deep learning approaches to inferring numerical coordinates for points of interest in an input image. Existing convolutional neural network-based solutions to this problem either take a heatmap matching approach or regress to coordinates with a fully connected output layer. Neither of these approaches is ideal, since the former is not entirely differentiable, and the latter lacks inherent spatial generalization. We propose our differentiable spatial to numerical transform (DSNT) to fill this gap. The DSNT layer adds no trainable parameters, is fully differentiable, and exhibits good spatial generalization. Unlike heatmap matching, DSNT works well with low heatmap resolutions, so it can be dropped in as an output layer for a wide range of existing fully convolutional architectures. Consequently, DSNT offers a better trade-off between inference speed and prediction accuracy compared to existing techniques. When used to replace the popular heatmap matching approach used in almost all state-of-the-art methods for pose estimation, DSNT gives better prediction accuracy for all model architectures tested.

Figures (11)

• Figure 1: Comparison of coordinate regression model architectures. The arrows indicate inference (black) and gradient flow (dashed red).
• Figure 2: Spatial representations of an example neck location. Image (b) is a 2D Gaussian rendered at the ground truth location, whereas (c) is learned freely by a model.
• Figure 3: Coordinate calculation using the differentiable spatial to numerical transform (DSNT).
• Figure 4: When heatmap matching, it is possible for predictions to worsen despite the pixel-wise MSE improving.
• Figure 5: Heatmap appearance for models trained with different regularization terms (red = right wrist, blue = left wrist).