Multi-path sensory substitution device navigates the blind and visually impaired individuals

Abstract

With the rapid advancements in computer vision technology, real-time autonomous navigation systems for blind and visually impaired individuals (BVIs) leveraging scene understanding have become increasingly feasible. Remarkably, existing navigation systems demonstrate excellent performance in obstacle detection and directional guidance for BVIs. Nevertheless, they lack the spatial perception of the entire scene for BVIs to reach the autonomous navigation. To overcome this issue, we propose a novel computer vision-based method to generate Multiple Paths with Sensory Substitution Device (MP-SSD) system, aiming to effectively and conveniently provide key autonomous navigation information for BVIs in outdoor routes. The MP-SSD system combines potential navigation target detection, path planning, and 3D Semantic Scene Completion (SSC) techniques to develop a widely applicable environmental detection and sensory substitution device (SSD), that can cater to the practical requirements of BVIs. Specifically, MP-SSD can extract semantic and spatial information from the input RGB image through the three-dimensional SSC model, and complete the information of the invisible area of the scene, thereby identifying potential target points and planning the shortest navigation path. During the interaction process, the system provides multi-path prompts through spatial audio to ensure accurate guidance. Experimental analyses on experience feedback of BVIs indicate that MP-SSD can effectively help BVIs actively acquire valuable navigation information in outdoor environments, thereby enhancing their autonomous mobility.