ReHARK: Refined Hybrid Adaptive RBF Kernels for Robust One-Shot Vision-Language Adaptation

Abstract

The adaptation of large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks with extremely limited data -- specifically in the one-shot regime -- is often hindered by a significant "Stability-Plasticity" dilemma. While efficient caching mechanisms have been introduced by training-free methods such as Tip-Adapter, these approaches often function as local Nadaraya-Watson estimators. Such estimators are characterized by inherent boundary bias and a lack of global structural regularization. In this paper, ReHARK (Refined Hybrid Adaptive RBF Kernels) is proposed as a synergistic training-free framework that reinterprets few-shot adaptation through global proximal regularization in a Reproducing Kernel Hilbert Space (RKHS). A multistage refinement pipeline is introduced, consisting of: (1) Hybrid Prior Construction, where zero-shot textual knowledge from CLIP and GPT-3 is fused with visual class prototypes to form a robust semantic-visual anchor; (2) Support Set Augmentation (Bridging), where intermediate samples are generated to smooth the transition between visual and textual modalities; (3) Adaptive Distribution Rectification, where test feature statistics are aligned with the augmented support set to mitigate domain shifts; and (4) Multi-Scale RBF Kernels, where an ensemble of kernels is employed to capture complex feature geometries across diverse scales. Superior stability and accuracy are demonstrated through extensive experiments on 11 diverse benchmarks. A new state-of-the-art for one-shot adaptation is established by ReHARK, which achieves an average accuracy of 65.83%, significantly outperforming existing baselines. Code is available at https://github.com/Jahid12012021/ReHARK.

Figures (8)

• Figure 1: 1-Shot performance comparison across 11 benchmarks. The proposed ReHARK method (red line with star markers) consistently outperforms existing training-free adaptation baselines.
• Figure 2: The overall architecture of the proposed ReHARK framework. Visual features and ensembled text weights (enriched by GPT3) undergo non-linear rectification before entering the core adaptation module. The system combines a refined semantic prior (global logic) with a multi-scale RBF kernel path (local adaptation) to solve for optimal adaptation coefficients in closed form.
• Figure 3: Qualitative analysis of ReHARK 1-shot predictions across 11 benchmarks. Green labels indicate correct classifications, while red labels denote misclassifications. The model demonstrates high fidelity in diverse domains, including fine-grained objects and complex scenes.
• Figure 4: t-SNE visualization of the latent space clusters generated by ReHARK. The Multi-Scale RBF kernels effectively capture the local geometry of class distributions across 11 datasets, facilitating distinct separation even with a single support sample per class.
• Figure 5: Ablation study evaluating the impact of individual architectural components. Removing the Power Transform causes the most significant performance degradation ($65.32\%$), while the Full ReHARK configuration maintains robust accuracy ($65.75\%$) across all components.