An Algorithm for Calculating Apple Picking Direction Based on 3D Vision

Abstract

Research into autonomous (robotic) apple picking has not yet resolved the problem of finding the optimal picking orientation. Robotic picking efficiency, in terms of picking all available apples without loss or damage, remains low. This paper proposes a method of determining the optimal picking orientation relative to the target fruit and adjacent branches from the point cloud of the apple and the surrounding space. The picking mechanism is then able to avoid branches and accurately grasp the target apple in order to pick it. The apple is first identified by the YOLOv3 target detection algorithm, and a point cloud of the fruit and the space surrounding it is obtained. The random sample consensus algorithm RANSAC is used for sphere fitting, and the fruit is idealized as a sphere. RANSAC also idealizes the branch as a line that is fitted to the branch bearing the target apple in the point cloud around it. The distance between the line of the branch and the fruit centroid is constrained in fitting to ensure identification of the branch/line closest to the apple/sphere. The best apple picking orientation is determined from the positional relationship between the straight branch/line and the center of the apple/sphere. The performance of the algorithm was evaluated using apples with various orientations on growing trees. The average angle error between the calculated picking direction vector and the expected direction vector was 11.81°, and the standard deviation was 13.65°; 62.658% of the determinations erred by ≤10°, and 85.021% erred by ≤20°. The average time for estimating the orientation of an apple was 0.543 s. The accuracy and speed of the algorithm enabled the robotic picker to operate at a speed that matches that of a human apple picker.