Cooperative learning of Pl@ntNet's Artificial Intelligence algorithm: how does it work and how can we improve it?

TL;DR

It is demonstrated that estimating users' skills based on the diversity of their expertise enhances labelling performance, emphasize the synergy of human annotation and data filtering in improving AI performance for a refined training dataset and explore incorporating AI‐based votes alongside human input in the label aggregation.

Abstract

Deep learning models for plant species identification rely on large annotated datasets. The PlantNet system enables global data collection by allowing users to upload and annotate plant observations, leading to noisy labels due to diverse user skills. Achieving consensus is crucial for training, but the vast scale of collected data makes traditional label aggregation strategies challenging. Existing methods either retain all observations, resulting in noisy training data or selectively keep those with sufficient votes, discarding valuable information. Additionally, as many species are rarely observed, user expertise can not be evaluated as an inter-user agreement: otherwise, botanical experts would have a lower weight in the AI training step than the average user. Our proposed label aggregation strategy aims to cooperatively train plant identification AI models. This strategy estimates user expertise as a trust score per user based on their ability to identify plant species from crowdsourced data. The trust score is recursively estimated from correctly identified species given the current estimated labels. This interpretable score exploits botanical experts' knowledge and the heterogeneity of users. Subsequently, our strategy removes unreliable observations but retains those with limited trusted annotations, unlike other approaches. We evaluate PlantNet's strategy on a released large subset of the PlantNet database focused on European flora, comprising over 6M observations and 800K users. We demonstrate that estimating users' skills based on the diversity of their expertise enhances labeling performance. Our findings emphasize the synergy of human annotation and data filtering in improving AI performance for a refined dataset. We explore incorporating AI-based votes alongside human input. This can further enhance human-AI interactions to detect unreliable observations.

Figures (9)

• Figure 1: Pl@ntNet system of human-AI interaction for plant species recognition. Users take their plant observations in the Pl@ntNet application. A prediction is output by the AI model. Users can validate the prediction or propose another species. The whole votes collection is used to evaluate user expertise (see \ref{['alg:plantnet_algorithm']}) and actively revise observations identifications.
• Figure 2: Weight function in \ref{['eq:weight_func']} used to map the number of identified species to a trust score in the Pl@ntNet label aggregation strategy. A new user starts with a weight of $f(0)=f(1)=\gamma\simeq 0.74$. The user confidence threshold $\theta_{\text{conf}}=2$ requires a user to have identified at least $n_u=8$ species to become self-validating. The parameters $\alpha=0.5$, $\beta=0.2$ and $\gamma\simeq 0.74$ are used in practice.
• Figure 3: Log-scales distribution of the observations in the South-West European Flora subset from the Pl@ntNet database. Note that the (sub-)datasets introduced are nested: $\mathcal{D}_{\mathrm{SWE}} \supset \mathcal{D}_\text{expert} \supset \mathcal{D}_{\text{multiple votes}} \supset \mathcal{D}_{\text{disagreement}}$. $\mathcal{D}_{\text{expert}}$ and the following subsets contain observations that received at least one vote from one of the experts.
• Figure 4: Pl@ntNet activity summary in the SWE flora subset. (A): The majority of users have proposed a small number of observations and species. However, some users have proposed a large number of observations and species. (B): In a perfectly balanced dataset, the Lorenz curve would be the diagonal -- $50\%$ of the votes would be for $50\%$ of the observations. In practice, there is a high imbalance of the distribution of votes between observations -- $80\%$ of the observations are represented by $10\%$ of votes.
• Figure 5: Accuracy of the aggregation strategies w.r.t. the proportion of classes (species) retrieved on subsets with at least two votes -- either agreeing (A) or with at least one disagreeing vote (B). The Pl@ntNet aggregation is more accurate, especially in a highly ambiguous setting (B). The TwoThird data filter highly impacts how many classes are kept in the dataset and the overall accuracy in both settings. WAWA and MV perform similarly with a benefit for WAWA when skill evaluation is needed.