Vision-Based Early Fault Diagnosis and Self-Recovery for Strawberry Harvesting Robots
Meili Sun, Chunjiang Zhao, Lichao Yang, Hao Liu, Shimin Hu, Ya Xiong
TL;DR
This work tackles stability and efficiency issues in strawberry-harvesting robots by introducing a vision-based fault-diagnosis and self-recovery framework. It combines SRR-Net, an end-to-end multi-task perception network for detection, segmentation, and ripeness estimation, with a relative error compensation mechanism and an early-abort strategy to address misalignment, empty grasps, and strawberry slippage. The approach achieves strong perception performance (e.g., ripeness MAE $=0.035$, FPS $=163.35$) and concrete gains in grasp accuracy and cycle-time reduction through compensation and aborts, demonstrating practical improvements for autonomous harvesting. The results on FaultData, GraspData, and SnapData indicate improved robustness and efficiency, with clear avenues for future gains through self-learning and adaptive perception.
Abstract
Strawberry harvesting robots faced persistent challenges such as low integration of visual perception, fruit-gripper misalignment, empty grasping, and strawberry slippage from the gripper due to insufficient gripping force, all of which compromised harvesting stability and efficiency in orchard environments. To overcome these issues, this paper proposed a visual fault diagnosis and self-recovery framework that integrated multi-task perception with corrective control strategies. At the core of this framework was SRR-Net, an end-to-end multi-task perception model that simultaneously performed strawberry detection, segmentation, and ripeness estimation, thereby unifying visual perception with fault diagnosis. Based on this integrated perception, a relative error compensation method based on the simultaneous target-gripper detection was designed to address positional misalignment, correcting deviations when error exceeded the tolerance threshold. To mitigate empty grasping and fruit-slippage faults, an early abort strategy was implemented. A micro-optical camera embedded in the end-effector provided real-time visual feedback, enabling grasp detection during the deflating stage and strawberry slip prediction during snap-off through MobileNet V3-Small classifier and a time-series LSTM classifier. Experiments demonstrated that SRR-Net maintained high perception accuracy. For detection, it achieved a precision of 0.895 and recall of 0.813 on strawberries, and 0.972/0.958 on hands. In segmentation, it yielded a precision of 0.887 and recall of 0.747 for strawberries, and 0.974/0.947 for hands. For ripeness estimation, SRR-Net attained a mean absolute error of 0.035, while simultaneously supporting multi-task perception and sustaining a competitive inference speed of 163.35 FPS.