NASR: Nonauditory speech recognition with motion sensors in head-mounted displays

Abstract

With the growing popularity of Virtual Reality (VR), people spend more and more time wearing Head-Mounted Display (HMD) for an immersive experience. HMD is physically attached on wearer’s head so that head motion can be tracked. We find it can also detect subtle movement of facial muscles which is strongly related to speech according to the mechanism of phonation. Inspired by this observation, we propose NonAuditory Speech Recognition (NASR). It uses motion sensor for recognizing spoken words. Different from most prior work of speech recognition using microphone to capture audio signal for analysis, NASR is resistant to acoustic noise of surroundings because of its nonauditory mechanism. Without using microphone, it consumes less power and requires no special permissions in most operating systems. Besides, NASR can be seamlessly integrated into existing speech recognition systems. Through extensive experiments, NASR can get up to 90.97% precision with 82.98% recall rate for speech recognition.