Low-Resource Vision Challenges for Foundation Models

TL;DR

This work tackles the under-explored problem of low-resource vision by introducing the LITE benchmark, which comprises circuit diagrams, historic maps, and mechanical drawings to expose data scarcity, fine-grained differences, and domain shift. It evaluates mainstream vision foundation models and finds them lacking in generalization to these constrained domains. To address this gap, the authors propose three simple baselines—generated data for scarcity, selective tokenization for fine-grained details, and attention for specialized domains—that can be added to frozen foundation models. The results show consistent improvements across six foundation models and three tasks, demonstrating the baselines’ complementary potential and highlighting ongoing challenges and directions for future work in low-resource vision.

Abstract

Low-resource settings are well-established in natural language processing, where many languages lack sufficient data for deep learning at scale. However, low-resource problems are under-explored in computer vision. In this paper, we address this gap and explore the challenges of low-resource image tasks with vision foundation models. We first collect a benchmark of genuinely low-resource image data, covering historic maps, circuit diagrams, and mechanical drawings. These low-resource settings all share three challenges: data scarcity, fine-grained differences, and the distribution shift from natural images to the specialized domain of interest. While existing foundation models have shown impressive generalizability, we find they cannot transfer well to our low-resource tasks. To begin to tackle the challenges of low-resource vision, we introduce one simple baseline per challenge. Specifically, we i) enlarge the data space by generative models, ii) adopt the best sub-kernels to encode local regions for fine-grained difference discovery and iii) learn attention for specialized domains. Experiments on our three low-resource tasks demonstrate our proposals already provide a better baseline than transfer learning, data augmentation, and fine-grained methods. This highlights the unique characteristics and challenges of low-resource vision for foundation models that warrant further investigation. Project page: https://xiaobai1217.github.io/Low-Resource-Vision/.

Figures (19)

• Figure 1: High-Resource vs Low-Resource Vision. High-resource vision focuses on images that can be collected at scale, have coarse-grained classes, and come from a general domain. We instead focus on low-resource vision tasks where data is scarce, has fine-grained differences, and comes from highly specialized domains.
• Figure 2: Low-Resource Image Transfer Evaluation Benchmark. Our three benchmark tasks are: (a) classifying circuit diagrams with the correct function, (b) retrieving the modern satellite map given an old map of a city, and (c) retrieving the mechanical drawing corresponding to a 3D photo of a component and vice versa.
• Figure 3: Generated Data for Data Scarcity. We augment images with generative models, obtaining images close to the input image where the label is preserved as well as more diverse images which break the label. We use label-preserving images in the task loss and augment the label-breaking images for use in a contrastive loss.
• Figure 4: Tokenization for Fine-Grained. We divide the original linear projection of a pre-trained foundation model into sub-kernels. These sub-kernels can be applied to smaller areas of the image patch to attend to fine-grained details. We learn a weighting to combine the resulting features into patch-level features.
• Figure 5: Attention for Specialized Domains. We learn a set of global attention maps with common attention patterns particular to the specialized domain such as vertical and horizontal directions for circuit diagrams. For each token, we crop the corresponding region from the global attention map according to the location.