Attention-based pose sequence machine for 3D hand pose estimation

Abstract

Most of the existing methods for 3D hand pose estimation are performed from a single depth map. In that case, the depth missing challenges from input frames caused by hand self-occlusions and imaging quality lead to multi-valued mapping phenomenon and sub-optimal model. In this paper, we proposed a novel recurrent architecture named Attention-based Pose Sequence Machine (APSM) to alleviate challenges by introducing temporal consistency. As for recurrent unit (RU), we extend traditional Gated Recurrent Unit (GRU) with 3D convolutional neural networks (CNNs) to handle voxelized inputs and features, and a novel RU named Deep Gated Recurrent Unit (DGRU) was proposed by rebuilding deeper gates based on GRU. To improve the model performance, a novel spatial attention mechanism denoted as Attention Model (AM) was proposed. Ablation experiments are designed to validate each contribution of our work, and experiments on two publicly available dataset show that our work outperforms state-of-the-art on hand pose estimation.