Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-time Distribution Learning Adapter for Cross-modal Visual Reasoning

Yi Zhang, Ce Zhang

TL;DR

This work proposes Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period and outperforms existing state-of-the-art methods on visual reasoning for human object interaction by large margins.

Abstract

Vision-Language Pre-Trained (VLP) models, such as CLIP, have demonstrated remarkable effectiveness in learning generic visual representations. Several approaches aim to efficiently adapt VLP models to downstream tasks with limited supervision, aiming to leverage the acquired knowledge from VLP models. However, these methods suffer from either introducing biased representations or requiring high computational complexity, which hinders their effectiveness in fine-tuning the CLIP model. Moreover, when a model is trained on data specific to a particular domain, its ability to generalize to uncharted domains diminishes. In this work, we propose Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period. Specifically, we estimate Gaussian distributions to model visual features of the few-shot support images to capture the knowledge from the support set. The cosine similarity between query image and the feature distribution of support images is used as the prediction of visual adapter. Subsequently, the visual adapter's prediction merges with the original CLIP prediction via a residual connection, resulting in the final prediction. Our extensive experimental results on visual reasoning for human object interaction demonstrate that our proposed TT-DNA outperforms existing state-of-the-art methods by large margins.

Test-time Distribution Learning Adapter for Cross-modal Visual Reasoning

TL;DR

This work proposes Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period and outperforms existing state-of-the-art methods on visual reasoning for human object interaction by large margins.

Abstract

Vision-Language Pre-Trained (VLP) models, such as CLIP, have demonstrated remarkable effectiveness in learning generic visual representations. Several approaches aim to efficiently adapt VLP models to downstream tasks with limited supervision, aiming to leverage the acquired knowledge from VLP models. However, these methods suffer from either introducing biased representations or requiring high computational complexity, which hinders their effectiveness in fine-tuning the CLIP model. Moreover, when a model is trained on data specific to a particular domain, its ability to generalize to uncharted domains diminishes. In this work, we propose Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period. Specifically, we estimate Gaussian distributions to model visual features of the few-shot support images to capture the knowledge from the support set. The cosine similarity between query image and the feature distribution of support images is used as the prediction of visual adapter. Subsequently, the visual adapter's prediction merges with the original CLIP prediction via a residual connection, resulting in the final prediction. Our extensive experimental results on visual reasoning for human object interaction demonstrate that our proposed TT-DNA outperforms existing state-of-the-art methods by large margins.
Paper Structure (9 sections, 11 equations, 2 figures, 3 tables)

This paper contains 9 sections, 11 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Background
  4. Test-time Distribution Learning Adapter (TT-DNA)
  5. TT-DNA with Fine Tuning (TT-DNA-F)
  6. Experiment
  7. Visual Reasoning on Bongard-HOI
  8. Ablation Study
  9. Conclusion

Figures (2)

  • Figure 1: Visual reasoning for human object interaction. We present the task description of the Bongard-HOI jiang2022bongard dataset. Please note that within the Bongard-HOI test set, there are a total of 6 positive examples, 6 negative examples, and a single query image.
  • Figure 2: Overview of the architecture of DNA. The figure presents (a) the overall architecture of DNA, (b) the distribution estimation and (c) the visual adapter inference process. The fire icon signifies that the parameters will undergo updates during the training process.