Data-Driven Worker Activity Recognition and Efficiency Estimation in Manual Fruit Harvesting

TL;DR

The paper addresses inefficiencies in manual strawberry harvesting by deploying instrumented picking carts to collect mass, location, and motion data, and developing a CNN-LSTM-based activity recognizer to label periods as 'Pick' or 'NoPick'. This enables precise estimation of picker efficiency and tray-fill times, validated on season-long data with high accuracy (mean ~95–96%) and an average active-harvest time of ~75%. Key contributions include a robust data annotation pipeline, a compact CNN-LSTM architecture with a U-shaped encoder and temporal LSTM modules, and a publicly released, richly annotated dataset for further research. The practical impact lies in improved labor management, optimized collection-station logistics, and enhanced yield-aware harvest planning in commercial strawberry production.

Abstract

Manual fruit harvesting is common in agriculture, but the amount of time pickers spend on non-productive activities can make it very inefficient. Accurately identifying picking vs. non-picking activity is crucial for estimating picker efficiency and optimising labour management and harvest processes. In this study, a practical system was developed to calculate the efficiency of pickers in commercial strawberry harvesting. Instrumented picking carts were developed to record the harvested fruit weight, geolocation, and cart movement in real time. These carts were deployed during the commercial strawberry harvest season in Santa Maria, CA. The collected data was then used to train a CNN-LSTM-based deep neural network to classify a picker's activity into "Pick" and "NoPick" classes. Experimental evaluations showed that the CNN-LSTM model showed promising activity recognition performance with an F1 score accuracy of over 0.97. The recognition results were then used to compute picker efficiency and the time required to fill a tray. Analysis of the season-long harvest data showed that the average picker efficiency was 75.07% with an estimation accuracy of 95.22%. Furthermore, the average tray fill time was 6.79 minutes with an estimation accuracy of 96.43%. When integrated into commercial harvesting, the proposed technology can aid growers in monitoring automated worker activity and optimising harvests to reduce non-productive time and enhance overall harvest efficiency.

Figures (7)

• Figure 1: Instrumented picking cart (iCarrito) developed by instrumenting traditional wire frame structure picking carts (Carrito). It consists of two load cells, a SwiftNav Piksi Multi GNSS unit, a control box with a Raspberry Pi 0W microcomputer, an SD card, control switches, and status LEDs.
• Figure 2: Field deployment of the iCarritos during commercial strawberry harvest in Santa Maria, CA
• Figure 3: Time series data collected from instrumented strawberry picking carts. The plots show harvested fruit mass; acceleration along the x-axis (red), y-axis (green), z-axis (blue); and velocity (magenta). Each sawtooth structure of the mass data represents full or partially full tray mass increments over time.
• Figure 4: The CNN-LSTM architecture used for picker activity classification. The U-shaped CNN encoder-decoder extracts hierarchical spatial features, while the LSTM layers capture temporal dependencies in the time-series data. The final convolution layers distinguish between "Pick" and 'NoPick" activities."
• Figure 5: Classification results of picker activity recognition during early (April 24, 2024, top) and peak (May 25, 2024, bottom) periods of harvest season. The square wave-like structures represent the model predictions. True positives (TP) and true negatives (TN) are marked in green, false positives (FP) in red, and false negatives (FN) in blue. The magenta colour indicates the harvest mass over time.