Estimating Appearance Models for Image Segmentation via Tensor Factorization

TL;DR

This work tackles the challenge of segmenting images into multiple regions without pre-specified appearance models. It introduces TEAM, a tensor-factorization–based method that estimates region appearances $\theta_1,\dots,\theta_K$ and their proportions $w$ from a single image by exploiting high-order color statistics via moments $\alpha,\beta,\gamma$ and a whitening/diagonalization procedure. When paired with a Markov Random Field graph-cut segmentation (TEAMSEG), the approach yields competitive or superior results on synthetic and real data, while automatically providing region proportions and avoiding filtering-based preprocessing. The method demonstrates robustness across IID-like and textured data, scales favorably in color and image size, and shows promise for applications in remote sensing and biomedical imaging, with future work aimed at linking to topic-modeling frameworks and improving memory efficiency.

Abstract

Image Segmentation is one of the core tasks in Computer Vision and solving it often depends on modeling the image appearance data via the color distributions of each it its constituent regions. Whereas many segmentation algorithms handle the appearance models dependence using alternation or implicit methods, we propose here a new approach to directly estimate them from the image without prior information on the underlying segmentation. Our method uses local high order color statistics from the image as an input to tensor factorization-based estimator for latent variable models. This approach is able to estimate models in multiregion images and automatically output the regions proportions without prior user interaction, overcoming the drawbacks from a prior attempt to this problem. We also demonstrate the performance of our proposed method in many challenging synthetic and real imaging scenarios and show that it leads to an efficient segmentation algorithm.

Figures (9)

• Figure 1: Example of an image with 3 regions. The set $(x, y, z)$ is an exemplar of $\mathcal{T}_r$, while $(x, y)$, $(y, z)$ and $(x, z)$ belong to $\mathcal{N}_r$.
• Figure 2: Ground-truth segmentation masks used in our synthetic experiments. Image frames are not part of the original images.
• Figure 3: Example of synthetic images generated from each generation process and for each segmentation mask used in our experiments. Image frames are not part of the original images.
• Figure 4: Varying the number of slices $\ell$ of $\gamma$ when estimating the appearance models of images from the four proposed ground-truth masks. The timing (\ref{['line:slices_gmm_et']}) and performance (\ref{['line:slices_gmm_db']}) of our method on GMM-generated images ($\sigma = 60$) are shown in dashed lines, whereas the solid lines (\ref{['line:slices_nogmm_et']} and \ref{['line:slices_nogmm_db']}) are for images generated from RAND. The lines show the average result of the estimations on 50 synthetic images of size $300 \times 300$ and with $L = 256$ colors.
• Figure 5: Varying the standard deviation of normal distributions on the images generated according to the GMM procedure under all available ground truth-masks. The estimation and segmentation results are shown in red and blue, respectively. Our method's results are plotted in solid lines, whereas the results arising from the EM-GMM algorithm are depicted in dashed lines. As in Figure \ref{['fig:vary_slices']}, the lines are the average results on 50 $300 \times 300$ images of $256$ colors. For segmentation, we set $\lambda = 1$ for both methods.

Theorems & Definitions (2)

• Proposition 1
• proof