WeedNet: A Foundation Model-Based Global-to-Local AI Approach for Real-Time Weed Species Identification and Classification

TL;DR

WeedNet introduces a global-scale, foundation-model approach for real-time weed species identification that leverages self-supervised pretraining on massive unlabeled data and targeted global-to-local fine-tuning with expert data. The model achieves 91.02% top-1 accuracy across 1,593 species, with strong local performance (97.38% across 84 Midwest weeds) and improved reliability via out-of-distribution detection and conformal prediction. The Global-to-Local strategy, coupled with edge-deployment capabilities and integration into PestIDBot, demonstrates practical viability for precision agriculture, robotic platforms, and ecological conservation. Findings highlight the importance of diverse global data, region-specific fine-tuning, and uncertainty quantification to address look-alike species, developmental-stage variation, and deployment constraints in real-world farming contexts.

Abstract

Early identification of weeds is essential for effective management and control, and there is growing interest in automating the process using computer vision techniques coupled with AI methods. However, challenges associated with training AI-based weed identification models, such as limited expert-verified data and complexity and variability in morphological features, have hindered progress. To address these issues, we present WeedNet, the first global-scale weed identification model capable of recognizing an extensive set of weed species, including noxious and invasive plant species. WeedNet is an end-to-end real-time weed identification pipeline and uses self-supervised learning, fine-tuning, and enhanced trustworthiness strategies. WeedNet achieved 91.02% accuracy across 1,593 weed species, with 41% species achieving 100% accuracy. Using a fine-tuning strategy and a Global-to-Local approach, the local Iowa WeedNet model achieved an overall accuracy of 97.38% for 85 Iowa weeds, most classes exceeded a 90% mean accuracy per class. Testing across intra-species dissimilarity (developmental stages) and inter-species similarity (look-alike species) suggests that diversity in the images collected, spanning all the growth stages and distinguishable plant characteristics, is crucial in driving model performance. The generalizability and adaptability of the Global WeedNet model enable it to function as a foundational model, with the Global-to-Local strategy allowing fine-tuning for region-specific weed communities. Additional validation of drone- and ground-rover-based images highlights the potential of WeedNet for integration into robotic platforms. Furthermore, integration with AI for conversational use provides intelligent agricultural and ecological conservation consulting tools for farmers, agronomists, researchers, land managers, and government agencies across diverse landscapes.

Figures (22)

• Figure 1: Examples of intra-species dissimilarity across developmental stages in common lambsquarters (Chenopodium album) and horseweed (Erigeron canadensis). The images show plant characteristics such as leaf shape, leaf margin, and change in plant morphology from seedling to fruiting stages. These variations within a single species across growth stages highlight the challenge for models to identify species consistently.
• Figure 2: Examples illustrating inter-species similarity challenges among various weed groups. The figure includes morphologically similar species from amaranth weeds, poisonous and non-poisonous species within the Apiaceae family, and closely related foxtail species. Despite toxicity or ecological function differences, these species exhibit similar phenotypic features, complicating accurate field identification.
• Figure 3: Overview of the Global Weed Model Development and Implementation. The Global Weed AI Model is trained on approximately 14 million images covering around 1600 plant species from the iNaturalist database. The end-to-end WeedNet pipeline was deployed on smartphones for real-time weed identification. The Global-to-Local approach enables fine-tuning on region-specific weed species using fewer images, reducing computational requirements while improving accuracy. Such local, region-specific lightweight models could identify weed images captured using robotic platforms such as UAVs and ground rovers, possibly deploying such models for high-throughput real-time weed identification.
• Figure 4: Example interface of the PestIDBot application on a common dandelion image. Users upload weed images for identification and return a prediction. The app also provides users with information on taxonomy, ecology, and identification traits.
• Figure 7: ROC curve over all possible threshold values. We chose a threshold value that provides the lowest FPR at 95% true positive rate (left). Box plot of energy values for in-distribution weed dataset (ID) and out-of-distribution (OOD) data. The red line is the threshold value used for separating ID and OOD data (center). Conformal prediction (CP) example:{Amaranthus palmeri (56%), Amaranthus spinosus (44%)} (right)