Creating a gesture-speech dataset for speech-based automatic gesture generation

Abstract

In our research, we recorded 298 min (1049 sentences) of speech audio data and the motion capture data of the accompanying gestures from two 25-year-old male participants aiming for future usage in deep learning concerning gesture and speech. The data was recorded in form of an interview, the participant explaining a topic prepared in advance, using a headset microphone and the motion capture software Motivë. The speech audio was stored in mp3, and the motion data was stored in bvh, related as data from the same sentence. We aimed to mainly acquire metaphoric gestures and iconic, as categorized by McNiel. For the categories of the recorded gestures, metaphoric gestures appeared the most, 68.41% of all gestures, followed by 23.73% beat gestures, 4.76% iconic gestures, and 3.11% deictic gestures.