Knock&Tap: Classification and Localization of Knock and Tap Gestures using Deep Sound Transfer Learning

Abstract

Gesture interaction is considered one of the promising approaches to control smart devices. In this paper, we present Knock&Tap, an audio-based approach that can perform gesture classification and gesture localization using deep transfer learning. Knock&Tap consists of a single 4-microphone array to record the sound of the user's knocking and tapping gestures and a wood/glass panel for knocking and tapping. Knock&Tap can be used in a situation or environment where vision-based gesture recognition is impossible due to the lighting condition or camera installation issue. Various experiments were conducted to validate the feasibility of Knock&Tap with 7 gesture types on both wood and glass panels. Our experimental results show that Knock&Tap predicts the gesture type and location with an accuracy of up to 97.24% and 92.05%, respectively.