Periodic Action Temporal Localization Method Based on Two-Path Architecture for Product Counting in Sewing Video

Abstract

Automatically product counting in the handmade process plays a vital role in the manufacturing industry, especially at the sewing industry. Nevertheless, there is currently a few methods to count the product number in the hand sewing process from surveillance video automatically. Due to the sewing procedure is a cyclical action, the product counting in hand sewing process is regarded as periodic action temporal localization and counting problem. In this paper, in order to solve this problem, we propose a novel two-path method, based on pose estimation and region-based convolutional neural network. The pose estimation method is used to obtain the trajectory information of human joint points, and the periodic action is located by detecting the periodic changes in joint trajectory. An effective two-threshold method is proposed to locate each action and count the number of periodic action from the trajectory information. To more accurately localization, we use a convolutional neural network to predict whether the workbench is empty or not. We fuse the results of joint trajectory and the status of the workbench to adjust the final periodic action localization and counting the number. To verify the proposed method, we built a new video database collected in the real sewing industry. The experimental results show that the proposed method is effective and constructive at periodic action localization and counting in the video for the sewing industry.