Interactive Hypergraph Visual Analytics for Exploring Large and Complex Image Collections

TL;DR

This work presents an end-to-end visual analytics approach for large, unannotated complex image collections by leveraging hypergraphs to represent overlapping relationships. It introduces a scalable hypergraph construction pipeline, a novel CES similarity measure for evaluating hypergraphs against ground truth, and a four-view, interactive visualization framework (Hyperedge List, Hyperedge Grid, Spatial Hypergraph Visualization, Hypergraph Matrix) that supports iterative exploration and targeted search. Empirical results show TEMI-based hypergraph construction outperforms traditional clustering, CES complements hNMI in quality assessment, and the multi-view UI remains usable on consumer hardware while guiding domain experts through real investigations. Collectively, these contributions enable efficient, interpretable analysis of tens of thousands of images and offer practical pathways for improving investigative workflows in domains such as forensics and accident analysis.

Abstract

Analyzing large complex image collections in domains like forensics, accident investigation, or social media analysis involves interpreting intricate, overlapping relationships among images. Traditional clustering and classification methods fail to adequately represent these complex relationships, particularly when labeled data or suitable pre-trained models are unavailable. Hypergraphs effectively capture overlapping relationships, but to translate their complexity into information and insights for domain expert users visualization is essential. We propose an interactive visual analytics approach specifically designed for the construction, exploration, and analysis of hypergraphs on large-scale complex image collections. Our core contributions include: (1) a scalable pipeline for constructing hypergraphs directly from raw image data, including a similarity measure to evaluate constructed hypergraphs against a ground truth, (2) interactive visualization techniques that integrate spatial hypergraph representations, interactive grids, and matrix visualizations, enabling users to dynamically explore and interpret relationships without becoming overwhelmed and disoriented, and (3) practical insights on how domain experts can effectively use the application, based on evaluation with real-life image collections. Our results demonstrate that our visual analytics approach facilitates iterative exploration, enabling domain experts to efficiently derive insights from image collections containing tens of thousands of images.

Figures (12)

• Figure 1: The system, with the Hyperedge list (top left), Grid visualization with intersecting hyperedges (top right), Spatial Hypergraph visualization (bottom left) and the Hypergraph matrix visualization (bottom right).
• Figure 2: Zooming into the Spatial Hypergraph view, the user can see and select the individual image nodes. The image nodes are placed based on the internal hyperedge UMAP projection. Images closer together are likely to be more similar.
• Figure 3: Typical workflow with our method for a CIC (in this case, MIC). The expert would first construct the hypergraph (1), then select (2) and visualize a hyperedge from the Hyperedge list using the Hyperedge Grid (3) and give it a label based on its content (4). Once the expert has given some structure by labeling, the Hypergraph matrix can be used to see if there are any interesting overlaps (5) between the cockpit and the fuselage hyperedge. The intersecting images can then be displayed (6) and the expert may see there are indeed images that depict the fuselage of the cockpit specifically. The expert can then decide to make a new hyperedge for this (7) and visualize the new hyperedge on the Spatial Hypergraph (8) to find additional candidates for this hyperedge. Using the lasso tool (9) these can be displayed on the grid and any relevant images can be added to the hyperedge (10).
• Figure 4: Some examples of the images in the accident investigation image collections.
• Figure 5: Results for various combinations of $t$ and $k$ for CUB200 and MIC, using either FCM or TEMI to construct the hypergraph and hNMI or CES to evaluate the constructed hypergraph against a ground truth hypergraph. The results show that TEMI outperforms FCM and that CES is more robust to over-segmentation than hNMI. FCM did not finish with $k=4096$