Hierarchy-Aware Fine-Tuning of Vision-Language Models

TL;DR

The paper addresses adapting Vision-Language Models to hierarchical label spaces by introducing two losses that enforce vertical and horizontal consistency in a shared embedding space. Tree-Path KL Divergence ($L_{TP-KL}$) aligns predicted taxonomic paths with ground-truth paths, while Hierarchy-Sibling Smoothed Cross-Entropy ($L_{HiSCE}$) smooths predictions among siblings; both are integrated within a Low-Rank Adaptation (LoRA) fine-tuning framework to update few parameters. Empirical results across four hierarchical benchmarks show consistent improvements in Full-Path Accuracy and reductions in Tree-based Inconsistency Error with minimal parameter overhead, validating the approach and its complementarity. The work provides a practical, taxonomy-aware adaptation strategy for VLMs with potential impact in biodiversity, medical imaging, and other domains requiring structured label understanding, formalized by the objective $L_{total}=L_{CE}+\u03bb_{1}L_{TP-KL}+L_{HiSCE}$ and a hierarchy-aligned embedding manifold.

Abstract

Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.

Figures (6)

• Figure 1: Overview of the proposed hierarchy-aware fine-tuning framework for Vision-Language Models.
• Figure 2: Illustration of hierarchy-sibling smoothing at a single taxonomy level. The ground-truth class B retains probability $1-\epsilon$, its sibling classes $C$ and $D$ share the remaining mass $\epsilon$ uniformly, and all non-sibling classes ($F$, $G$) receive $0$. This corresponds to one row of the smoothing matrix $T^{(l)}$ in Equation (1).
• Figure 3: Overview of the Tree-Path KL Divergence computation.
• Figure 4: Performance comparison across datasets under different TP-KL weights $\lambda$: FPA (top) and TICE (bottom).
• Figure 5: Visualization of text embeddings before and after joint TP-KL + HiSCE optimization on 2 datasets (Butterfly-200 and CUB-200-2011). Left: t-SNE projections, where each point denotes a label embedding colored by its hierarchy level. Right: dendrograms visualizing hierarchical grouping, where branch colors indicate clustering structure.