Vision Graph Prompting via Semantic Low-Rank Decomposition

TL;DR

This work tackles efficient adaptation of Vision GNNs (ViG) to downstream tasks by exploiting the graph-structured semantics of images. It introduces Vision Graph Prompting (VGP), a semantic low-rank prompting framework with three components—SeLo-Graph, SeLo-Edge, and SeLo-Node—that capture global and local semantics while keeping most of the ViG backbone frozen. The key finding is that semantic information in vision graphs resides in a low-rank subspace; leveraging this via low-rank prompts yields substantial accuracy gains (averaging ~5.0% on vision benchmarks) with ~94.6% fewer trainable parameters and minimal compute overhead, outperforming Transformer-based prompts on ViG. The approach also generalizes to traditional graph tasks, underscoring the broader applicability of semantic low-rank prompting for graph-structured data. This work presents a practical direction for parameter-efficient, graph-aware visual adaptation with broad implications for graph-based vision and beyond.

Abstract

Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditional grid or sequence-based representations. To efficiently adapt ViG to downstream tasks, parameter-efficient fine-tuning techniques like visual prompting become increasingly essential. However, existing prompting methods are primarily designed for Transformer-based models, neglecting the rich topological relationships among nodes and edges in graph-based representations, limiting their capacity to model complex semantics. In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures. Our core insight reveals that semantically connected components in the graph exhibit low-rank properties. Building on this observation, we introduce a semantic low-rank prompting method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-grained semantic dependencies. Extensive experiments demonstrate our method significantly improves ViG's transfer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our code is available at https://github.com/zhoujiahuan1991/ICML2025-VGP.

Figures (5)

• Figure 1: Visualization of ViG graph structures using PCA and t-SNE. This figure illustrates the graph structures in ViG, where semantically related components exhibit shared principal features and form a single compact cluster in t-SNE embeddings. The visualization underscores the low-rank nature of semantic information.
• Figure 2: Observation on ViG graph structures and principal components. The center patch (red star) is randomly selected from the primary semantic region, with neighboring patches (blue dot) linked via edges. PCA is applied to patches from the same image group (a, b), mapping the first three components to the RGB color channels. Despite variations in shape, texture, and color, the ViG graph effectively connects semantically related object parts. These connected regions share common principal components, demonstrating a low-rank structure. Background regions are filtered out by thresholding the first principal component.
• Figure 3: Pipeline of Vision Graph Prompting via semantic low-rank decomposition. Specifically, we design: (i) SeLo-Graph Prompt as trainable virtual nodes that dynamically form edges with existing nodes to capture global semantic dependencies, (ii) SeLo-Edge Prompt with low-rank decomposition to enhance feature propagation between semantically connected nodes, and (iii) SeLo-Node Prompt to refine fine-grained semantic information while preserving local intrinsic details.
• Figure 4: Ablation on blending factors $\alpha$ and $\beta$. The red star marks the best result.
• Figure 5: Visualization of ViG graph structures, the first three major PCA components, and the average distribution of PCA component coefficient magnitudes.