Manifold-Preserving Superpixel Hierarchies and Embeddings for the Exploration of High-Dimensional Images

TL;DR

This paper presents a superpixel hierarchy for high-dimensional images that takes the high-dimensional attribute manifold into account during construction and enables consistent exploration of high-dimensional images in both image and attribute space.

Abstract

High-dimensional images, or images with a high-dimensional attribute vector per pixel, are commonly explored with coordinated views of a low-dimensional embedding of the attribute space and a conventional image representation. Nowadays, such images can easily contain several million pixels. For such large datasets, hierarchical embedding techniques are better suited to represent the high-dimensional attribute space than flat dimensionality reduction methods. However, available hierarchical dimensionality reduction methods construct the hierarchy purely based on the attribute information and ignore the spatial layout of pixels in the images. This impedes the exploration of regions of interest in the image space, since there is no congruence between a region of interest in image space and the associated attribute abstractions in the hierarchy. In this paper, we present a superpixel hierarchy for high-dimensional images that takes the high-dimensional attribute manifold into account during construction. Through this, our method enables consistent exploration of high-dimensional images in both image and attribute space. We show the effectiveness of this new image-guided hierarchy in the context of embedding exploration by comparing it with classical hierarchical embedding-based image exploration in two use cases.

Figures (17)

• Figure 1: Image Hierarchies: Classical image-space-based hierarchies, like image pyramids, progressively blur and subsample the image (a). Attribute-space-based hierarchies, as used in hierarchical DR methods (b), ignore the image space entirely and only aim to preserve manifold structure of the attribute space. A pixel, as highlighted in the hierarchy, might actually be represented by multiple landmarks in abstraction levels. And, in turn, a landmark can represent scattered pixels. Combined, this complicates an image-based exploration of the high-dimensional space. Our superpixel hierarchy (c) combines both image layout and attribute space manifold structure. and color two superpixels on the middle abstraction level, the respective coarse-grained vertices in the attribute graph and corresponding embedding points.
• Figure 2: Graph structures: 4-connected image graph $\mathcal{I}$ (left) and attribute-based graph $\mathcal{G}$ (right). Different neighborhoods of the same node 1 are highlighted in grey.
• Figure 3: Method overview: Each image pixel is associated with a high-dimensional attribute vector (top left). (1) We compute a neighborhood graph $\mathcal{G}$, whose vertices correspond to pixels and edges are based on attribute similarities. The data level embedding is computed from this graph, like in, e.g., t-SNE. (2) We compute a feature vector per vertex, describing the local graph structure, using random walks. (3) For the next abstraction level, vertices in the attribute graph $\mathcal{G}$ are merged with the most similar neighbor in $\mathcal{I}$. (4) The new vertex retains all outgoing connections of the merged vertices. All vertex features are added and re-normalized. (5) This is repeated for each vertex. (6) Similarities between merged vertices are used for creating embeddings where each point corresponds to one superpixel (right).
• Figure 4: Indian Pines Exploration: (a) a false color image based on data channels 20 (587 nm, red), 76 (1090 nm, green) and 130 (1591 nm, blue) with a ROI marked in red. (b) and (c) show the 4th abstraction level embedding of our superpixel embedding and HSNE, respectively; (d) and (e) highlight the superpixels and landmarks which correspond to the ROI. (f) and (g) show refined embeddings of the highlighted subsets on a lower abstraction level. (h) and (i) recolor a cutout of the image based on the refined embeddings using an overlaid 2D colormap Finally, (j) indicates the pixels in the full images which are represented by the HSNE refinement, whereas our superpixel refinement extends only slightly around the ROI.
• Figure 5: Non-exact refinement as discussed in \ref{['sec:method:ComputeEmbeddings']}. Here, using $\gamma = 0.01$ can improve the embedding structure by including additional superpixels not in the ROI (c.f. \ref{['fig:IndianPinesLargeExploration:f']}).