GPFS: A graph-based human pose forecasting system for smart home with online learning

Abstract

Forecasting human poses given a sequence of historical pose frames has several important applications, especially in the domain of smart home safety. Recently, computer vision-based human pose forecasting has made a breakthrough using deep learning technology. However, to implement a practical system deployed on an IoT edge environment, there are still two issues to be addressed. First, existing methods on pose forecasting fail to model the coherent structural information of connected human joints and thus cannot achieve satisfactory prediction accuracy, especially for long-term predictions. Second, a general and static pre-trained prediction model may not perform well in the deployment environment due to the visual domain shift problem. In this article, we propose a hybrid cloud-edge system called GPFS to solve those issues. Specifically, we first introduce a novel graph convolutional neural network (GCN)-based sequence-to-sequence learning method, which enhances the sequence encoder by using a graph to represent both the spatial and temporal connections of the human joints in the input frames. The GCN improves the forecasting accuracy by capturing the motion pattern of each joint as well as the correlations among different human joints over time. Second, to address the domain shift issue and protect data privacy, we extend the system to perform online learning on the IoT edge to adapt the cloud trained general model with online collected on-site domain data. Extensive evaluation on Human 3.6M and Penn Action datasets demonstrates the superiority of our proposed system.