Reconstructing what you said: Text inference using smartphone motion

Abstract

Smartphones and tablets are becoming ubiquitous within our connected lives and as a result these devices are increasingly being used for more and more sensitive applications, such as banking. The security of the information within these sensitive applications is managed through a variety of different processes, all of which minimise the exposure of this sensitive information to other potentially malicious applications. This paper documents experiments with the 'zero-permission' motion sensors on the device as a side-channel for inferring the text typed into a sensitive application. These sensors are freely accessible without the phone user having to give permission. The research was able to, on average, identify nearly 30 percent of typed bigrams from unseen words, using a very small volume of training data, which was less than the size of a tweet. Given the natural redundancy in language this performance is often enough to understand the phrase being typed. We found that large devices were typically more vulnerable, as were users who held the device in one hand whilst typing with fingers. Of those bigrams which were not correctly identified over 60 percent of the errors involved the space bar and nearly half of the errors are within two keys on the keyboard.