AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment

TL;DR

This work presents AgriCLIP, a vision-language foundational model dedicated to the domain of agriculture and livestock, and proposes a large-scale dataset, named ALive, that leverages customized prompt generation strategy to overcome the scarcity of expert annotations and a training pipeline that integrates both contrastive and self-supervised learning.

Abstract

Capitalizing on vast amount of image-text data, large-scale vision-language pre-training has demonstrated remarkable zero-shot capabilities and has been utilized in several applications. However, models trained on general everyday web-crawled data often exhibit sub-optimal performance for specialized domains, likely due to domain shift. Recent works have tackled this problem for some domains (e.g., healthcare) by constructing domain-specialized image-text data. However, constructing a dedicated large-scale image-text dataset for sustainable area of agriculture and livestock is still open to research. Further, this domain desires fine-grained feature learning due to the subtle nature of the downstream tasks (e.g, nutrient deficiency detection, livestock breed classification). To address this we present AgriCLIP, a vision-language foundational model dedicated to the domain of agriculture and livestock. First, we propose a large-scale dataset, named ALive, that leverages customized prompt generation strategy to overcome the scarcity of expert annotations. Our ALive dataset covers crops, livestock, and fishery, with around 600,000 image-text pairs. Second, we propose a training pipeline that integrates both contrastive and self-supervised learning to learn both global semantic and local fine-grained domain-specialized features. Experiments on diverse set of 20 downstream tasks demonstrate the effectiveness of AgriCLIP framework, achieving an absolute gain of 7.8\% in terms of average zero-shot classification accuracy, over the standard CLIP adaptation via domain-specialized ALive dataset. Our ALive dataset and code can be accessible at \href{https://github.com/umair1221/AgriCLIP/tree/main}{Github}.

Figures (3)

• Figure 1: Overview of our proposed framework, consisting of the ALive dataset and the AgriCLIP training pipeline, designed to integrate both global semantic and local fine-grained domain-specialized features. The ALive is an image-text dataset for agriculture and livestock domain that is constructed by leveraging images and their metadata to prompt GPT-4, generating customized text for each image. The AgriCLIP training pipeline consists of Semantic Feature Learning, where contrastive learning is utilized to train image and text encoders; Fine-Grained Feature Learning, using a self-supervised approach to train the vision encoder; and Cross-Model Alignment, aligning vision encoders from the previous stages to enable zero-shot generalization.
• Figure 2: Example images from ALive dataset, including various crops (such as dates, crop diseases, and plant genera), diverse fish species, and samples from the livestock domain. More examples are in the appendix.
• Figure 3: Some more examples of the ALive dataset