Pocketear: An assistive sound classification system for hearing-impaired

Abstract

This paper describes the design and operation of an assistive system called PocketEAR which is primarily targeted towards hearing-impaired users. It helps them with orientation in acoustically active environments by continuously monitoring and classifying the incoming sounds and displaying the captured sound classes to the users. The environmental sound recognizer is designed as a two-stage deep convolutional neural network classifier (consists of the so-called superclassifier and a set of the so-called subclassifiers) fed with sequences of MFCC vectors. It is wrapped in a distributed client-server system where the sound capturing in terrain, (pre)processing and displaying of the classification results are performed by instances of a mobile client application, and the actual classification and maintenance are carried out by two co-operating servers. The paper discusses in details the architecture of the environmental sound classifier as well as the used task-specific sound processing.