The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings

TL;DR

This paper investigates whether incorporating recurrence into a state-of-the-art image segmentation model improves performance under noise and few-shot conditions. It evaluates three energy-based recurrence paradigms—Self-organizing maps, Conditional Random Fields, and Hopfield networks—added to an EfficientUnet++ backbone on artificial shapes and a medical CAD dataset. Across experiments, recurrence yields improvements on synthetic data under noise and scarce data, but fails to consistently surpass the baseline on medical data, highlighting the challenge of energy-function design and data-domain transfer. The results suggest a nuanced role for recurrence, with potential in combination or hybrid approaches rather than as a standalone replacement for feed-forward segmentation.

Abstract

The biological brain has inspired multiple advances in machine learning. However, most state-of-the-art models in computer vision do not operate like the human brain, simply because they are not capable of changing or improving their decisions/outputs based on a deeper analysis. The brain is recurrent, while these models are not. It is therefore relevant to explore what would be the impact of adding recurrent mechanisms to existing state-of-the-art architectures and to answer the question of whether recurrency can improve existing architectures. To this end, we build on a feed-forward segmentation model and explore multiple types of recurrency for image segmentation. We explore self-organizing, relational, and memory retrieval types of recurrency that minimize a specific energy function. In our experiments, we tested these models on artificial and medical imaging data, while analyzing the impact of high levels of noise and few-shot learning settings. Our results do not validate our initial hypothesis that recurrent models should perform better in these settings, suggesting that these recurrent architectures, by themselves, are not sufficient to surpass state-of-the-art feed-forward versions and that additional work needs to be done on the topic.

Figures (15)

• Figure 1: This figure illustrates the fact that a system with feedback connections is able to change its decision with time. For that, at least one cycle is needed in the computational graph.
• Figure 2: An illustration of recurrency, where the dynamics are governed by gradient descent, which implies the minimization of an energy function $E$ w.r.t. a state $s$.
• Figure 3: A self-organizing map square lattice.
• Figure 4: Given a filter, we can produce a grid-like graph with more than one connected component, where each component will have pixels from only one class.
• Figure 5: The original image, figure A, is split into equal sized patches, figure A. These patches are then used as tokens for the memory retrieval process of the modern Hopfield network.