MV-Match: Multi-View Matching for Domain-Adaptive Identification of Plant Nutrient Deficiencies

TL;DR

This work proposes an approach that leverages multiple camera views in the source and target domain for unsupervised domain adaptation and achieves state-of-the-art results on both datasets compared to other unsupervised domain adaptation methods.

Abstract

An early, non-invasive, and on-site detection of nutrient deficiencies is critical to enable timely actions to prevent major losses of crops caused by lack of nutrients. While acquiring labeled data is very expensive, collecting images from multiple views of a crop is straightforward. Despite its relevance for practical applications, unsupervised domain adaptation where multiple views are available for the labeled source domain as well as the unlabeled target domain is an unexplored research area. In this work, we thus propose an approach that leverages multiple camera views in the source and target domain for unsupervised domain adaptation. We evaluate the proposed approach on two nutrient deficiency datasets. The proposed method achieves state-of-the-art results on both datasets compared to other unsupervised domain adaptation methods. The dataset and source code are available at https://github.com/jh-yi/MV-Match.

Figures (6)

• Figure 1: Illustration of our multi-view setting. Related views are expected to share the same nutrient status, both in the labeled source and the unlabeled target domain. We do not assume that each view contains exactly the same plant, but very closely located plants with the same nutrient status, which makes the data capturing in an open field very simple.
• Figure 2: Proposed approach for unsupervised domain adaptation. Given multiple views of a crop in the labeled source domain (top) and the unlabeled target domain (bottom), a random query image is sampled from the source and the target domain (center of the green and blue box). The Similarity-guided View Mining (SgVM) module then computes the normalized mutual information between each query-view pair to select the top n dissimilar views of the same crop (red dashed rectangles). From these two sets, we randomly select a second image for each query image. We then apply weak or strong data augmentation to the four images, i.e., the two query images and their corresponding view pair images, and feed them to a shared model for predicting nutrient deficiencies. While the prediction of the query image of the source domain is supervised by the ground-truth label, the other predictions are enforced to be consistent with the corresponding view pair. For this, the query image with weak augmentation is always considered as a reference prediction, both for the source and target domain.
• Figure 3: Saliency visualization (a) without adaptation and (b) with adaptation. For each case, we show the original image, saliency map obtained by guided backpropagation springenberg2015striving, and by GradCAM++ chattopadhay2018grad.
• Figure 4: Example images. Columns 1-6: -N, -P, -K, -B, -S, ctrl; row 1-3: 21 June 2022, 04 July 2022, and 20 July 2022.
• Figure 5: Example images. Column 1-6: -N, -P, -K, -B, -S, ctrl; row 1-3: 12 May 2023, 18 May 2023, and 24 May 2023.