HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

Abstract

Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a "one-size-fits-all" architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D -> D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.

Figures (4)

• Figure 1: Chronological Base-to-Novel Generalization. Novel Accuracy (blue) and Harmonic Mean (red) across 11 datasets. HeBA (Ours) sets a new state-of-the-art with 78.62% Novel Accuracy and 81.35% HM.
• Figure 2: Overview of the HeBA framework. We keep the pre-trained CLIP radford2021learning backbone frozen (indicated by ) and inject lightweight, modality-specific adapters. Left: Enriched text prompts combine standard handcrafted templates with fine-grained LLM descriptions (CuPL) to enhance semantic representation. Top: The Text Adapter employs a Bottleneck linear architecture to preserve semantic integrity while compressing dimensions. Bottom: The Visual Adapter explicitly captures spatial inductive biases using $3 \times 3$ depthwise convolutions (DW-Conv). Key Innovation: Unlike prior methods, the up-projection layers utilize Active Kaiming Initialization to provide immediate gradient flow, driving rapid feature adaptation from the first iteration and mitigating zero-gradient stagnation.
• Figure 3: Model-level Inductive Bias Integration in HeBA.Left: Parallel adapters are inserted into the frozen Transformer block (MSA and MLP) to learn residual corrections. Right: The Text Adapter utilizes a linear bottleneck ($D \rightarrow D/4$) to preserve semantics, while the Visual Adapter employs $3\times3$ Depthwise Convolutions to enforce spatial locality. Crucially, up-projections use Kaiming Initialization to actively stimulate learning, modulated by a dynamic scaling factor $s$.
• Figure 4: Fine-grained Performance Comparison on Structure-Sensitive Datasets. We report the Novel Accuracy (top) and Harmonic Mean (bottom) on four challenging benchmarks: DTD (textures) cimpoi2014describing, EuroSAT (satellite imagery) helber2019eurosat, Oxford Flowers (fine-grained) nilsback2008automated, and UCF101 (actions) soomro2012ucf101. These domains require capturing local spatial correlations, which standard MLP-based adapters gao2023clip often neglect. HeBA (Red bars) consistently outperforms the previous state-of-the-art LwEIB yang2025learning (Blue bars) and other baselines (Gray bars).