Higher-order interactions of multi-layer prompt

TL;DR

MSGCOT tackles the limitation of single-granularity prompts by introducing a multi-scale prompt chain-of-thought framework for graphs. It builds a low-rank, multi-scale coarsening network to generate hierarchical thoughts used as prompts, and employs a backtracking mechanism to progressively integrate coarse-to-fine information, optimized by a combined loss $L_{final}=L_{ds}+\alpha L_r$. Empirically, MSGCOT yields significant improvements on node and graph classification across eight datasets, with strong benefits in few-shot settings, and ablations confirm the value of multi-scale prompts and the backtracking strategy. The approach also demonstrates parameter efficiency, encoder-agnostic performance, and applicability to heterophilous graphs, highlighting the practical impact of modeling higher-order interactions among multi-layer prompts.

Abstract

The "pre-train, prompt" paradigm has successfully evolved in representation learning. While current prompt-tuning methods often introduce learnable prompts, they predominantly treat prompts as isolated, independent components across different network layers. This overlooks the complex and synergistic higher-order interactions that exist between prompts at various hierarchical depths, consequently limiting the expressive power and semantic richness of the prompted model. To address this fundamental gap, we propose a novel framework that explicitly models the Higher-order Interactions of Multi-layer Prompt. Our approach conceptualizes prompts from different layers not as separate entities, but as a cohesive system where their inter-relationships are critical. We design an innovative interaction module that captures these sophisticated, non-linear correlations among multi-layer prompts, effectively modeling their cooperative effects. This allows the model to dynamically aggregate and refine prompt information across the network's depth, leading to a more integrated and powerful prompting strategy. Extensive experiments on eight benchmark datasets demonstrate that our method, by leveraging these higher-order interactions, consistently surpasses state-of-the-art prompt-tuning baselines. The performance advantage is particularly pronounced in few-shot scenarios, validating that capturing the intricate interplay between multi-layer prompts is key to unlocking more robust and generalizable representation learning.

Figures (8)

• Figure 1: The comparison of graph prompt tuning methods in the existing studies.
• Figure 2: The framework of the proposed MSGCOT.
• Figure 3: Impact of shots number analysis.
• Figure 4: The sensitivity of the coarsening rate.
• Figure 5: The sensitivity of the coarsening layer number.