Auditory sensation aided image perception: Integration of somesthetic sense of fingertip position

Abstract

A color image presentation system with sound via touch-panel interface was developed and some perceptual characteristics with the system were studied in this paper. Each subject actively operates his/her fingertip on the touch panel interface, and the system returns syllable sounds corresponding to the colors nearby the fingertip position. Without sight, he/she perceives images through the sound information together with the fingertip positional sense. The authors carried out some preliminary psychophysical experiments on the human perceptual characteristics in converting speech sounds into image patterns and on the fingertip positional sense. Based on the results of the preliminary experiments, the authors developed a color image presentation system. It was confirmed through experiments using the system that the subjects were able to perceive the shapes of simple closed polygonal figures within 2 or 3 minutes.