Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology-Aware CLIP Few-Shot Learning

Dazhi Huang

TL;DR

The paper addresses the challenge of efficiently adapting large vision-language models to downstream tasks in few-shot regimes by preserving pre-trained knowledge while learning task-specific cues. It introduces a topology-aware fine-tuning approach that integrates Representation Topology Divergence (RTD) into the Task Residual framework, freezing the base encoders and training a lightweight residual classifier, with the total objective $L_{total} = L_{CE} + \lambda L_{RTD}$. Across six diverse benchmarks, RTD-TR yields consistent improvements over strong baselines, demonstrating the value of aligning multi-scale topology between visual and textual embeddings. This topology-guided regularization enhances few-shot generalization and offers a principled, efficient path to leverage topological information in VLM adaptation, while acknowledging computational costs and pointing to avenues for future refinement.

Abstract

Efficiently adapting large Vision-Language Models (VLMs) like CLIP for few-shot learning poses challenges in balancing pre-trained knowledge retention and task-specific adaptation. Existing methods often overlook valuable structural information within the VLM's latent space. We introduce a topology-aware tuning approach integrating Representation Topology Divergence (RTD) into the Task Residual (TR) framework. By explicitly aligning the topological structures of visual and text representations using a combined RTD and Cross-Entropy loss, while freezing base VLM encoders, our method enhances few-shot performance. We optimize only lightweight Task Residual parameters, effectively leveraging topological information. Across 6 diverse benchmark datasets, our approach demonstrates significant gains, achieving an average accuracy improvement of 1-2\% over relevant baseline methods in few-shot settings. This work presents an effective strategy to boost VLM few-shot capabilities by incorporating topological alignment.

Topology-Aware CLIP Few-Shot Learning

TL;DR

The paper addresses the challenge of efficiently adapting large vision-language models to downstream tasks in few-shot regimes by preserving pre-trained knowledge while learning task-specific cues. It introduces a topology-aware fine-tuning approach that integrates Representation Topology Divergence (RTD) into the Task Residual framework, freezing the base encoders and training a lightweight residual classifier, with the total objective . Across six diverse benchmarks, RTD-TR yields consistent improvements over strong baselines, demonstrating the value of aligning multi-scale topology between visual and textual embeddings. This topology-guided regularization enhances few-shot generalization and offers a principled, efficient path to leverage topological information in VLM adaptation, while acknowledging computational costs and pointing to avenues for future refinement.

Abstract

Efficiently adapting large Vision-Language Models (VLMs) like CLIP for few-shot learning poses challenges in balancing pre-trained knowledge retention and task-specific adaptation. Existing methods often overlook valuable structural information within the VLM's latent space. We introduce a topology-aware tuning approach integrating Representation Topology Divergence (RTD) into the Task Residual (TR) framework. By explicitly aligning the topological structures of visual and text representations using a combined RTD and Cross-Entropy loss, while freezing base VLM encoders, our method enhances few-shot performance. We optimize only lightweight Task Residual parameters, effectively leveraging topological information. Across 6 diverse benchmark datasets, our approach demonstrates significant gains, achieving an average accuracy improvement of 1-2\% over relevant baseline methods in few-shot settings. This work presents an effective strategy to boost VLM few-shot capabilities by incorporating topological alignment.
Paper Structure (16 sections, 5 equations, 4 figures)

This paper contains 16 sections, 5 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Persistent Homology
  4. Comparing Topological Structures
  5. Cross-Barcode for Comparing Data Manifolds
  6. Representation Topology Divergence (RTD)
  7. Differentiable RTD for Topology-Preserving Representations
  8. Related Work
  9. Topological Data Analysis in Deep Learning
  10. Methodology
  11. Task Residual Framework with Topological Regularization
  12. Training Strategy for Topological Alignment
  13. Experiments and Results
  14. Experimental Settings
  15. Performance Analysis
  16. ...and 1 more sections

Figures (4)

  • Figure 1: A filtration of a complex from a point cloud of four points.
  • Figure 2: Graphs $\mathcal{G}^{w\leq\alpha}$, $\mathcal{G}^{\hat{w}\leq\alpha}$ and $\mathcal{G}^{min(w,\hat{w})\leq\alpha}$ with edges not in $\mathcal{G}^{w\leq\alpha}$ colored in blue
  • Figure 3: The proposed topologically-aware few-shot learning framework. The training loss combines Representation Topology Divergence ($L_{RTD}$) between visual and adapted text embeddings with Cross-Entropy ($L_{CE}$) to update only the Task Residual.
  • Figure 4: The performance comparison of our proposed topologically-aware method (RTD-TR) against baseline approaches (CLIP-Adapter, TaskRes, Tip-Adapter-F) on few-shot learning tasks. Results include accuracy for 1, 2, 4, 8, and 16 shots per class across 6 benchmark datasets.