Braille Tactile-to-Auditory Conversion System Based on Self-Learning Flexible Tactile Sensor Array With Attention-Mechanism Model

Abstract

This article introduces a portable, instant Braille tactile-to-auditory conversion recognition system to address the challenges of high learning costs and recognition difficulties of Braille characters. We contrive a self-learning high-density flexible tactile sensor array in conjunction with an attention mechanism neural network, for the first time, to design a tactile recognition model capable of full-area scanning for Braille characters. A 16 × 16 sensor array consists of 256 ultrasmall micro-electro-mechanical system (MEMS) silicon pressure sensors with a pitch of 0.65 mm. Furthermore, we develop a tactile-to-auditory conversion system integrated with a Raspberry Pi 4B, enabling real-time conversion from mechanical stimuli to electrical signals and then to audio signals. Using self-developed application software, real-time interaction between the system and users is achieved. In comparison with various neural network models, the proposed model achieves the English alphabet recognition rate of up to 0.97 and the digits recognition rate of up to 0.96. Through the validation and testing experiments, our system demonstrates an accuracy rate of up to 0.95 and a precision rate of up to 0.98, with time efficiency on CPU and GPU reaching 1.3 and 0.5 s, respectively. The phonic Braille recognition system facilitates interactive feedback between tactility and audition, thereby granting visually impaired individuals an elevated level of independence, autonomy, and unrestricted access to information, with the added advantage of zero learning costs.