A Nonlocal Graph-PDE and Higher-Order Geometric Integration for Image Labeling

TL;DR

An entropy-regularized difference-of-convex-functions (DC) decomposition of this potential is devised and it is shown that the basic geometric Euler scheme for integrating the assignment flow is equivalent to solving the G-PDE by an established DC programming scheme.

Abstract

This paper introduces a novel nonlocal partial difference equation (G-PDE) for labeling metric data on graphs. The G-PDE is derived as nonlocal reparametrization of the assignment flow approach that was introduced in \textit{J.~Math.~Imaging \& Vision} 58(2), 2017. Due to this parameterization, solving the G-PDE numerically is shown to be equivalent to computing the Riemannian gradient flow with respect to a nonconvex potential. We devise an entropy-regularized difference-of-convex-functions (DC) decomposition of this potential and show that the basic geometric Euler scheme for integrating the assignment flow is equivalent to solving the G-PDE by an established DC programming scheme. Moreover, the viewpoint of geometric integration reveals a basic way to exploit higher-order information of the vector field that drives the assignment flow, in order to devise a novel accelerated DC programming scheme. A detailed convergence analysis of both numerical schemes is provided and illustrated by numerical experiments.

Figures (18)

• Figure 1.1: Summary of results. Starting point (Section \ref{['sec:Assignment-Flow']}) is a particular formulation of the assignment flow ODE (top) that represents the Riemannian gradient descent of a functional $J$ (left). The first main contribution of this paper is an equivalent alternative representation of the assignment flow equation in terms of a partial difference equation on the underlying graph (right), with a nonlocal data-driven diffusion term in divergence form and further terms induced by the information-geometric approach to the labeling problem. The second major contribution concerns a DC-decomposition of the nonconvex functional $J$ (bottom) and a novel accelerated minimization algorithm using a second-order tangent space parametrization of the assignment flow.
• Figure 2.1: Schematic visualization of a nonlocal boundary.Left: A bounded open domain $\Omega \subset \mathbb{R}^2$ with local boundary $\partial \Omega$ overlaid by the grid $\mathbb{Z}^2$. Right: A bounded open domain $\Omega$ with nonlocal boundary (light gray color). Nodes and , respectively, are vertices on the graph $\mathcal{V}$ and on the interaction domain $\mathcal{V}_{\mathcal{I}}^{\alpha}$ given by \ref{['eq:Interaction_Dom']}.
• Figure 2.2: Inference of label assignments via assignment flows.Center column: Application task of assigning data to prototypes in a metric space. Right column: Overview of the geometric approach Astrom:2017ac. The data are represented by the distance matrix $D_{\mathcal{X}}$ and by the likelihood vector field $L(W)$ on the assignment manifold $\mathcal{W}$. The similarity vectors $S(W)(x)$, determined through geometric averaging of the likelihood vectors, drive the assignment flow whose numerical geometric integration result in spatially coherent and unique label assignment to the data. Left column: Alternative equivalent reformulation of the assignment flow Savarino:2019ab which separates (i) the influence of the data that only determine the initial point of the flow (cf. \ref{['eq:S-flow-S']}), and (ii) the influence of the parameters $\Omega$ that parametrize the vector field which drives the assignment flow. This enables to derive the novel nonlocal geometric diffusion equation in Section \ref{['sec:Non_Local_PDE']}.
• Figure 2.3: Two image labeling scenarios demonstrating the influence of nonlocal regularization. Top: Application of assignment flows to a 3D medical imaging problem for segmenting the human retina (see Sitenko:2021vu a detailed exposition). (a): A B-scan from a 3D OCT-volume showing a section of the human retina that is corrupted by speckle noise. (b): The corresponding ground truth labeling with ordered retina layers. (c): Output from a Resnet that serves as the distance matrix \ref{['eq:Dis_F']}. (d): Result of applying assignment flow with local neighborhoods given by a 3D seven point stencil. (e): Labeling obtained with nonlocal uniform neighborhoods of size $|\mathcal{N}| = 11 \times 11 \times 11$. Increasing the connectivity leads to more accurate labeling that satisfy the ordering constraint depicted in (b). Bottom: Labeling of noisy data by assignment flows with data-driven parameters $\Omega$ determined by nonlocal means Buades:2010aa using patches of size $7\times 7$ pixels. (f): Synthetic image with thin repetitive structure. (g): Severly corrupted input image to be labeled with $\mathcal{X}^{\ast} = \{, , \}$. (h),(i): Labeling by the assignment flow that was regularized with neighborhood sizes $|\mathcal{N}|=3\times 3$ and $|\mathcal{N}|=11\times 11$, respectively. Enlarging the neighborhood size $|\mathcal{N}|$ increases labeling accuracy.
• Figure 3.1: Labeling through the nonlocal geometric assignment flow with uniform parametrization \ref{['eq:uniform-params']} and neighborhood size $|\mathcal{N}|=7$. (a) Ground truth with 31 labels. (b) Noisy input data used to evaluate \ref{['eq:S-flow-S']} and \ref{['eq:Non_Local_PDE']}, respectively. (c) Labeling returned when using the zero nonlocal Dirichlet boundary condition. (d) Labeling returned when using the non-zero nonlocal Dirichlet boundary condition (uniform extension to the interaction domain). The close-up views show differences close to the boundary, whereas the results in the interior domain are almost equal.

Theorems & Definitions (30)

• Remark 2.1
• Remark 2.2: Regularization
• Lemma 3.1
• Remark 3.2: Comments
• Proposition 3.3
• Proposition 3.4: nonlocal balance law of assignment flows
• Proposition 5.1
• Lemma 5.2
• Proposition 5.3
• Remark 5.4: directly related work