CT-Bound: Robust Boundary Detection From Noisy Images Via Hybrid Convolution and Transformer Neural Networks

TL;DR

The quantitative analysis shows that CT-Bound outperforms the previous best algorithms in edge detection on very noisy images and increases the edge detection accuracy of FoJ-based methods while having a 3-time speed improvement.

Abstract

We present CT-Bound, a robust and fast boundary detection method for very noisy images using a hybrid Convolution and Transformer neural network. The proposed architecture decomposes boundary estimation into two tasks: local detection and global regularization. During the local detection, the model uses a convolutional architecture to predict the boundary structure of each image patch in the form of a pre-defined local boundary representation, the field-of-junctions (FoJ). Then, it uses a feed-forward transformer architecture to globally refine the boundary structures of each patch to generate an edge map and a smoothed color map simultaneously. Our quantitative analysis shows that CT-Bound outperforms the previous best algorithms in edge detection on very noisy images. It also increases the edge detection accuracy of FoJ-based methods while having a 3-time speed improvement. Finally, we demonstrate that CT-Bound can produce boundary and color maps on real captured images without extra fine-tuning and real-time boundary map and color map videos at ten frames per second.

Figures (9)

• Figure 1: Boundary detection from noisy images. Compared to a variety of models mia2023boundariesfu2023practicalzhou2023treasurezamir2022restormerxie2015holisticallycanny1986computationalpu2022edtersu2021pixelverbin2021field, ours robustly detects the boundaries even when they are visually challenging to discriminate.
• Figure 2: Field-of-junction (FoJ) representation verbin2021field.
• Figure 3: Network architecture of CT-Bound. The architecture consists of two stages. The initialization stage contains shared-weights convolutional neural networks that output the FoJ representation of every image patch. The refinement stage contains a transformer encoder that simultaneously refines all per-patch FoJ representations. Finally, the framework combines all per-patch FoJ representations together to output the global boundary map and the color map.
• Figure 4: Effect of refinement and boundary selection. (a-b) An input noisy image and the corresponding clean image from the MS COCO dataset lin2014microsoft. (c-d) The color map and boundary map before the refinement stage. (e-f) The color map and boundary map after the refinement stage. Noisy edge estimations are removed in refinement, and the color map is smoother.
• Figure 5: Qualitative comparison of noisy images synthesized from BSDS500 arbelaez2010contour and NYUDv2 silberman2012indoor datasets with photon level $\alpha_{\text{test}}=2$. The proposed method shows robustness to the high noise level, while other methods fail to produce accurate boundaries. Additionally, ours can detect faint boundaries that are visually invisible.