Pose Sequence-Aware Generative Adversarial Network for Augmenting Skeleton Sequences to Improve Cerebral Palsy Detection by Deep Learner

Abstract

In medical analysis, early detection and diagnosis of cerebral palsy (CP) are highly helpful in stimulating brain cells and alleviating the adverse impacts of the disease. In the detection of CP, diagnostic tools such as magnetic resonance imaging (MRI) and general movements assessment (GMA) have emerged. In the past years, most studies have focused on GMA-based CP detection based on the different pose estimation methods. Amongst, the part affinity field (PAF) or OpenPose was one of the well-known methods for estimating the skeletal images from the RGB-D videos of infant general movements, which was used for CP detection. But, the estimated skeletal images were very few and it was difficult to annotate a large infant movement dataset. Therefore, in this article, a pose sequence-aware generative adversarial network (PS-GAN)-based data augmentation method is proposed that creates high-quality skeleton images for CP detection. First, long-range dependencies in continuous frames are acquired by self-attention and the dense graph is pruned to achieve efficient training. Then, spatial joints and temporal characteristics are encoded into the PS-GAN using the graph convolutional network (GCN) to map noises to high-quality skeleton images. Besides, the PS-GAN structure selection problem is defined as a Markov decision process (MDP) and solved using the new reinforcement learning (RL) to choose the optimal PS-GAN structure, which achieves effective generation. Further, the created skeleton images are used to train the convolutional neural network (CNN) with a softmax classifier and detect CP. At last, an extensive experiment is conducted on the MINI-RGBD, babyPose and motion infant analysis (MIA) databases. The results show that the PS-GAN-CNN achieves 92.2%, 92.5% and 92% accuracy for MINI-RGBD, babyPose and MIA databases in contrast with the PredictMed, fully connected network (FCNet), CNN-long shortterm memory (LSTM) and knowledge-based recurrent neural network (KBRNN) methods.