Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

TL;DR

This work tackles the gap between NLP-style zero-shot generalization and computer vision by introducing Explanatory Instructions that describe the true objective of vision tasks as linguistic transformations. It builds the Dataset of Explanatory CV Tasks (DECVT), split into Terminological-based and Explanatory-based components, to train a vanilla token-based AR vision-language model. The experiments show promising instruction-level zero-shot generalization and, under data-curation strategies, emerging vision-task-level generalization on unseen tasks, validating the potential of explanatory descriptions to unify vision tasks. While limitations remain—particularly around tokenizer-text alignment and some task types—the approach offers a principled pathway toward more flexible, universally applicable vision systems.

Abstract

Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input $\to$ explanatory instruction $\to$ output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.

Figures (57)

• Figure 1: (a) Early CV models are designed to handle discrete vision tasks. (b) Recent VLMs use terminological instructions (i.e., terminological task definitions), e.g., "semantic segmentation" and "pose map". (c) We propose Explanatory Instructions to explain CV tasks' objective and construct the dataset of Explanatory CV Tasks. We train the model via this dataset. (d) The trained model showcases instruction-level zero-shot capabilities. (e) By omitting certain human-defined vision tasks in the training dataset (cf. Sec. \ref{['sec:exp_tzs']}), we demonstrate promising vision task-level zero-shot capabilities.
• Figure 2: Examples of terminological-based vision tasks, e.g., holistically nested edge detection.
• Figure 3: Examples of explanatory-based vision tasks.
• Figure 4: Framework of our vanilla token-based VLM method.
• Figure 5: Examples of instruction-level zero-shot capabilities.