Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

TL;DR

This work tackles the problem of selecting the best Vision–Language Model (VLM) for a target task in few-shot settings by grounding transferability in model-specific internal dynamics. It introduces a target-specific task representation based on layer conductance of the visual encoder and derives an asymmetric transfer metric, Directional Conductance Divergence (DCD), via entropy-regularized alignment to measure how well a source task covers target-critical blocks. By converting DCD into a similarity distribution and aggregating source-task ranks, the method predicts target-model rankings without direct evaluation, requiring only unlabeled images. Empirically, the approach outperforms state-of-the-art baselines across 48 models and 21 datasets, achieving a 14.7% improvement in NDCG@5 and demonstrating strong data efficiency and architecture-aware transfer reasoning. The theoretical analysis supports the necessity of asymmetry under coverage semantics, underscoring the practical impact for scalable VLM selection in real-world deployments with limited data.

Abstract

While open sourced Vision-Language Models (VLMs) have proliferated, selecting the optimal pretrained model for a specific downstream task remains challenging. Exhaustive evaluation is often infeasible due to computational constraints and data limitations in few shot scenarios. Existing selection methods fail to fully address this: they either rely on data-intensive proxies or use symmetric textual descriptors that neglect the inherently directional and model-specific nature of transferability. To address this problem, we propose a framework that grounds model selection in the internal functional dynamics of the visual encoder. Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment. Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks. This allows for predicting target model rankings by aggregating source task ranks without direct inference. Experimental results on 48 VLMs across 21 datasets demonstrate that our method outperforms state-of-the-art baselines, achieving a 14.7% improvement in NDCG@5 over SWAB.

Figures (7)

• Figure 1: How we measure proxy reliability. We construct a performance gap matrix $G$ with entries $G_{ij}=|\mathrm{Acc}_i-\mathrm{Acc}_j|$, and a proxy gap matrix $G'$ from either semantic distances or conductance-induced distances.
• Figure 2: Model-to-Model Correlation Matrix in Task Perception. High correlations within architectural families and low correlations across them indicate model-specific task relationships.
• Figure 3: Sensitivity to $\eta$ and $\gamma$. Heatmaps of selection performance when varying $\eta$ (softmax temperature for target-conditioned block importance) and $\gamma$ (softmin temperature for DCD-based source weighting). All values are averaged over 10 random seeds. Performance is stable across a wide range.
• Figure 4: Impact of Source Task Sample Size ($N_{src}$). With a fixed single-shot target representation ($N_{tgt}=1$), performance improves rapidly and saturates around $N_{src}=25$, demonstrating that our method requires minimal data to construct the source reference library.
• Figure 6: Boxplots of main results over 10 random runs. Each box summarizes the distribution across runs for three methods (ModelGPT, swab, ours).

Theorems & Definitions (11)

• Definition 5.1: Salient Set
• Proposition 5.4: Impossibility of Symmetry
• Remark 5.5
• Theorem 3.1
• proof
• proof
• Lemma 4.1: Tail Mass Bound
• proof
• Proposition 4.2: DCD as a Set-Restricted Relaxation
• proof