Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Fine-grained Visual Understanding in VLMs through Text-Only Training

Dasol Choi, Guijin Son, Soo Yong Kim, Gio Paik, Seunghyeok Hong

TL;DR

This work tackles the data-intensive nature of vision-language models by evaluating text-only training as a pathway to improve fine-grained visual understanding. By training two 7B open-source VLMs on two domains—the butterfly species domain and a Korean cultural understanding domain—using both image-text and text-only data, the study shows that text-only training can achieve comparable or superior performance with substantially reduced compute and energy usage. The authors present strong evidence against data contamination through image-free evaluations and demonstrate robustness across both simple visual recognition and more complex reasoning tasks. The findings suggest text-driven visual representations can efficiently guide visual understanding, offering a scalable and resource-efficient alternative for VLM adaptation in resource-constrained settings.

Abstract

Visual-Language Models (VLMs) have become a powerful tool for bridging the gap between visual and linguistic understanding. However, the conventional learning approaches for VLMs often suffer from limitations, such as the high resource requirements of collecting and training image-text paired data. Recent research has suggested that language understanding plays a crucial role in the performance of VLMs, potentially indicating that text-only training could be a viable approach. In this work, we investigate the feasibility of enhancing fine-grained visual understanding in VLMs through text-only training. Inspired by how humans develop visual concept understanding, where rich textual descriptions can guide visual recognition, we hypothesize that VLMs can also benefit from leveraging text-based representations to improve their visual recognition abilities. We conduct comprehensive experiments on two distinct domains: fine-grained species classification and cultural visual understanding tasks. Our findings demonstrate that text-only training can be comparable to conventional image-text training while significantly reducing computational costs. This suggests a more efficient and cost-effective pathway for advancing VLM capabilities, particularly valuable in resource-constrained environments.

Improving Fine-grained Visual Understanding in VLMs through Text-Only Training

TL;DR

This work tackles the data-intensive nature of vision-language models by evaluating text-only training as a pathway to improve fine-grained visual understanding. By training two 7B open-source VLMs on two domains—the butterfly species domain and a Korean cultural understanding domain—using both image-text and text-only data, the study shows that text-only training can achieve comparable or superior performance with substantially reduced compute and energy usage. The authors present strong evidence against data contamination through image-free evaluations and demonstrate robustness across both simple visual recognition and more complex reasoning tasks. The findings suggest text-driven visual representations can efficiently guide visual understanding, offering a scalable and resource-efficient alternative for VLM adaptation in resource-constrained settings.

Abstract

Visual-Language Models (VLMs) have become a powerful tool for bridging the gap between visual and linguistic understanding. However, the conventional learning approaches for VLMs often suffer from limitations, such as the high resource requirements of collecting and training image-text paired data. Recent research has suggested that language understanding plays a crucial role in the performance of VLMs, potentially indicating that text-only training could be a viable approach. In this work, we investigate the feasibility of enhancing fine-grained visual understanding in VLMs through text-only training. Inspired by how humans develop visual concept understanding, where rich textual descriptions can guide visual recognition, we hypothesize that VLMs can also benefit from leveraging text-based representations to improve their visual recognition abilities. We conduct comprehensive experiments on two distinct domains: fine-grained species classification and cultural visual understanding tasks. Our findings demonstrate that text-only training can be comparable to conventional image-text training while significantly reducing computational costs. This suggests a more efficient and cost-effective pathway for advancing VLM capabilities, particularly valuable in resource-constrained environments.

Paper Structure

This paper contains 22 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Datasets
  4. Training Dataset
  5. Evaluation Datasets
  6. Models
  7. Experimental Results
  8. Main Results
  9. Analysis by Question Types
  10. Visual Recognition Tasks (Type 1)
  11. Complex Reasoning Tasks (Type 2)
  12. Evidence Against Data Contamination
  13. Resource Comparison
  14. Conclusion
  15. Acknowledgements
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Illustration of our approach comparing direct visual exposure and language-driven understanding. Left: Traditional image-text paired training with direct visual exposure. Right: Our text-only training approach using detailed descriptions. Bottom: Example evaluation setting where both approaches are tested on the same visual understanding tasks.
  • Figure 2: Resource efficiency comparison between Image+Text and Text-only training approaches.