Pix2Poly: A Sequence Prediction Method for End-to-end Polygonal Building Footprint Extraction from Remote Sensing Imagery

TL;DR

Pix2Poly tackles end-to-end extraction of polygonal building footprints from aerial imagery by predicting vertex sequences and learning vertex connectivity to form ring graphs. It introduces a two-module architecture: a Vertex Sequence Detector (transformer encoder-decoder) for corner sequences and an Optimal Matching Network (Sinkhorn-based) for connectivity, enabling differentiable, end-to-end training without rasterization losses. Across several challenging datasets (INRIA, SpaceNet, WHU, Massachusetts Roads, SpaceNet 2), Pix2Poly achieves state-of-the-art polygon quality metrics and demonstrates strong road-network extraction with lower model complexity (~31.9M parameters) and fast inference (~18 ms per image). The approach simplifies the training pipeline by avoiding post-processing polygonization and non-differentiable NMS, and is readily applicable to both building footprints and road graphs.

Abstract

Extraction of building footprint polygons from remotely sensed data is essential for several urban understanding tasks such as reconstruction, navigation, and mapping. Despite significant progress in the area, extracting accurate polygonal building footprints remains an open problem. In this paper, we introduce Pix2Poly, an attention-based end-to-end trainable and differentiable deep neural network capable of directly generating explicit high-quality building footprints in a ring graph format. Pix2Poly employs a generative encoder-decoder transformer to produce a sequence of graph vertex tokens whose connectivity information is learned by an optimal matching network. Compared to previous graph learning methods, ours is a truly end-to-end trainable approach that extracts high-quality building footprints and road networks without requiring complicated, computationally intensive raster loss functions and intricate training pipelines. Upon evaluating Pix2Poly on several complex and challenging datasets, we report that Pix2Poly outperforms state-of-the-art methods in several vector shape quality metrics while being an entirely explicit method. Our code is available at https://github.com/yeshwanth95/Pix2Poly.

Figures (13)

• Figure 1: Polygons predicted by SOTA methods on the INRIA test split. Segmentation approaches ictnet & indirect methods Girard_2021_CVPRXu2022AccuratePM suffer from poor quality at building corners & edges. In contrast, Pix2Poly can generate high-quality building footprint polygons.
• Figure 2: Overview of the Pix2Poly architecture.signify the losses for (i) vertex detection $\mathcal{L}_{detection}^{v}$, and (ii) permutation matrices $\mathcal{L}_{permutation}$. The outputs are displayed as . The vertex detection network is indicated with & the optimal matching network which learns the connections between the detected vertices is depicted as . Ground truth data is shown as .
• Figure 3: Vertex Sequence Detector. The Vertex Sequence Detector is an encoder-decoder transformer network that predicts a sequence of discrete building corner coordinates using the aerial image as input. depicts the losses for vertex detection $\mathcal{L}_{detection}$. The outputs are displayed as . The ground truth corner coordinates & sequence used in the training phase are shown as .
• Figure 4: Optimal Matching Network. The optimal matching network produces a connection score matrix, $S\in\mathbb{R}^{N_v \times N_v}$ for every possible vertex pair in the predicted sequence from the Vertex Sequence Detector. We ensure high-quality predicted polygons by enforcing path consistency; the counter-clockwise score matrix is the transpose of the clockwise score matrix. Optimal vertex assignments are generated from the predicted score matrices in the form of a permutation matrix.
• Figure 5: Qualitative comparisons. Examples of predicted building polygons from the INRIA test set. We compare with FFL Girard_2021_CVPR ($1^{st}$ row), HiSup Xu2022AccuratePM ($2^{nd}$ row) and ours ($3^{rd}$ row).