DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning

TL;DR

This work tackles few-shot learning by introducing DVLA-RL, a framework that enables hierarchical vision–language alignment across low- and high-level semantics. It combines Dual-level Semantic Construction (DSC), which generates discriminative attributes and coherent descriptions via LLMs, with Adaptive RL-gated Attention (RLA), which dynamically balances self- and cross-attention for cross-modal fusion across network layers. The approach yields state-of-the-art results across nine benchmarks spanning general, fine-grained, and cross-domain FSL, and its ablations confirm the complementary value of dual-level semantics and adaptive fusion in reducing semantic hallucination and improving generalization. DVLA-RL demonstrates strong robustness to different LLMs and maintains efficient training and inference, highlighting practical benefits for low-data visual recognition tasks.

Abstract

Few-shot learning (FSL) aims to generalize to novel categories with only a few samples. Recent approaches incorporate large language models (LLMs) to enrich visual representations with semantic embeddings derived from class names. However, they overlook progressive and adaptive alignment between vision and language from low-level to high-level semantics, resulting in limited semantic gains. To address these challenges, we propose Dual-level Vision-Language Alignment with Reinforcement Learning gating (DVLA-RL), which consists of Dual-level Semantic Construction (DSC) and RL-gated Attention (RLA). Specifically, DSC conditions LLMs on both class names and support samples to generate discriminative attributes, progressively selects the most relevant ones, and then synthesizes them into coherent class descriptions. This process provides complementary low-level attributes and high-level descriptions, enabling both fine-grained grounding and holistic class understanding. To dynamically integrate dual-level semantics along with the visual network layers, RLA formulates cross-modal fusion as a sequential decision process. A lightweight policy trained with episodic REINFORCE adaptively adjusts the contributions of self-attention and cross-attention to integrate textual and visual tokens. As a result, shallow layers refine local attributes and deep layers emphasize global semantics, enabling more precise cross-modal alignment. This achieves class-specific discrimination and generalized representations with merely a few support samples. DVLA-RL achieves new state-of-the-art performance across nine benchmarks in three diverse FSL scenarios.

Figures (4)

• Figure 1: Overview of the proposed DVLA-RL framework. Dual-level Semantic Construction (DSC) extracts visual attributes with LLMs, progressively selects the most discriminative ones, and synthesizes them into class descriptions. Adaptive RL-Gated Attention (RLA) integrates these dual-level semantics with visual tokens, dynamically balancing self- and cross-attention between visual and textual tokens across layers for hierarchical and adaptive vision–language alignment.
• Figure 2: The effect of different selection strategies and Top-$k$ number.
• Figure 3: T-SNE visualization on novel classes from four datasets.
• Figure 4: Meta-training algorithm of the proposed DVLA-RL.