Visual Prompt-Agnostic Evolution

TL;DR

This work addresses unstable training dynamics in Visual Prompt Tuning (VPT) for Vision Transformers by diagnosing shallow-layer stagnation and deep-layer gradient oscillations. It introduces Prompt-Agnostic Evolution (PAE), a two-part framework combining Modal Pre-Alignment (MPA) for task-aligned frequency shortcuts and Koopman-Lyapunov discrete dynamical system (KLD) to couple prompts across layers with a shared operator, reinforced by a Lyapunov regularizer for stability. Empirically, PAE yields faster convergence (average $1.41\times$ speedup) and accuracy gains of $1$–$3\%$ across 25 datasets, while remaining prompt-agnostic and backbone-free at inference. Analyses show flatter loss landscapes, improved localization via Grad-CAM, and coherent cross-layer prompt evolution evidenced by CKA, confirming the benefits of viewing VPT as a controlled dynamical process. The approach scales across ViT and Swin backbones and extends to segmentation tasks, offering a practical enhancement for lightweight downstream adaptation.

Abstract

Visual Prompt Tuning (VPT) adapts a frozen Vision Transformer (ViT) to downstream tasks by inserting a small number of learnable prompt tokens into the token sequence at each layer. However, we observe that existing VPT variants often suffer from unstable training dynamics, characterized by gradient oscillations. A layer-wise analysis reveals that shallow-layer prompts tend to stagnate early, while deeper-layer prompts exhibit high-variance oscillations, leading to cross-layer mismatch. These issues slow convergence and degrade final performance. To address these challenges, we propose Prompt-Agnostic Evolution ($\mathtt{PAE}$), which strengthens vision prompt tuning by explicitly modeling prompt dynamics. From a frequency-domain perspective, we initialize prompts in a task-aware direction by uncovering and propagating frequency shortcut patterns that the backbone inherently exploits for recognition. To ensure coherent evolution across layers, we employ a shared Koopman operator that imposes a global linear transformation instead of uncoordinated, layer-specific updates. Finally, inspired by Lyapunov stability theory, we introduce a regularizer that constrains error amplification during evolution. Extensive experiments show that $\mathtt{PAE}$ accelerates convergence with an average $1.41\times$ speedup and improves accuracy by 1--3% on 25 datasets across multiple downstream tasks. Beyond performance, $\mathtt{PAE}$ is prompt-agnostic and lightweight, and it integrates seamlessly with diverse VPT variants without backbone modification or inference-time changes.

Figures (11)

• Figure 1: (a) Comparison of accuracy and convergence speed is shown in multiple VPT variants, including VPT, VPT+EMA, E2VPT, VFPT, and SA2VP. (b) Gradient oscillation (12 layers mean) is observed in multiple VPT variants, i.e., VPT, E2VPT, VFPT, and SA2VP. (c) VPT hierarchically exhibits shallow-layer (Layers 1--4) stagnation and deep-layer (Layers 9--12) oscillations.
• Figure 2: (a) $\mathtt{PAE}$ pipeline: $\mathtt{MPA}$ first initialize per-layer prompts. $\mathtt{KLD}$ then propagates prompts across layers via a shared Koopman operator, with a Lyapunov-style regularizer constraining error growth. (b) $\mathtt{MPA}$ pipeline: Frequency-domain transformations generate candidate masks, from which top ones are selected to build the initial prompt and propagate it across layers.
• Figure 3: Illustration of Koopman evolution. In standard VPT, prompts at different layers are independent. In contrast, $\mathtt{KLD}$ maps each layer’s prompt into a shared latent space, where a global shared Koopman operator $\mathbf{K}$ governs their evolution, enabling smooth cross-layer transitions.
• Figure 4: Loss landscape comparisons li2018visualizing show (left to right): 2-D loss contours, sharpness (max Hessian eigenvalue), anisotropy (Hessian condition number), and a 3-D surface colored by curvature.
• Figure 5: Grad-CAM visualizations at early training stages (epochs 5, 30, and 50) show that VPT+$\mathtt{PAE}$ rapidly concentrates attention on class-discriminative regions and stabilizes the saliency patterns, showing faster convergence than VPT.