Hypergraph Vision Transformers: Images are More than Nodes, More than Edges

TL;DR

HgVT introduces a hierarchical bipartite hypergraph within a Vision Transformer to capture higher-order semantic relationships while preserving computational efficiency. By dynamically constructing adjacency with cosine-based querying, using communication-pool attention across vertex and hyperedge streams, and enforcing semantic structure through diversity, population, and expert pooling, HgVT achieves strong ImageNet classification and retrieval results with isotropic architectures. The work provides thorough ablations, analyzes graph-quality metrics, and demonstrates practical advantages in semantic clustering and scalable retrieval, suggesting a path toward more efficient, semantically aware vision models. Overall, HgVT offers a principled framework for sparse, adaptive graph-based representations in vision, with potential for scalable deployment and self-supervised extensions.

Abstract

Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance on image classification and retrieval, positioning it as an efficient framework for semantic-based vision tasks.

Figures (33)

• Figure 1: Comparison of Graph Structures for different methods. Showing (a) CNNs, (b) Vision Transformers, (c) ViG with a KNN clustered GNN, (d) ViHGNN with clustered hyperedges, and (e) our proposed HgVT method. Strong group edges shown with solid lines; weak edges with dashed lines. Hyperedges shown with shaded regions; less dominant hyperedges with gray dashed regions.
• Figure 2: Comparison of (a) hypergraph and (b) equivalent bipartite representation from \ref{['fig:hyper_tree_vihgnn']}, showing five hyperedges.
• Figure 3: HgVT Architecture Diagram, composed of stacked HgVT blocks with adjacency matrix $\mathbf{A}$, vertex features ${\mathbf{X}^{(V)}}$, and hyperedge features $\mathbf{X}^{(E)}$. Edge attention flow is shown with gray arrows; input norms and residual adds are omitted for clarity.
• Figure 4: (a) Hyperedge Communication Pool Flow with edges $\mathcal{E}$ and member vertices $\mathcal{V}_e$; (b) Attention Sparsity (Mean and std) for HgVT-S on the ImageNet-1k Validation set.
• Figure 5: HgVT Architecture, composed of stacked HgVT blocks with adjacency matrix $\mathbf{A}$, vertex features $\mathbf{X}^{(V)}$, and hyperedge features $\mathbf{X}^{(E)}$. Pooling only applied to $\mathbf{X}^{(:iV)}$ and $\mathbf{X}^{(:vE)}$; edge attention flow shown with gray arrows; norms and residual omitted for clarity.