Interactive banknotes recognition for the visual impaired with wearable assistive devices

Abstract

In this paper, we develop a new system, named WVIAS (Wearable Vision Impaired Assistive System), using camera-based computer vision technology to recognize banknote in natural scene aim to help visually impaired people. WVIAS is made up of two mainly parts. In the front, there is a micro camera, set on the glass or mounted on the helmet, to acquire video sequence. In the back, a high performance portable computer is planted to run processing algorithm. To make the system robust to variety conditions including occlusion, rotation, scaling, cluttered background, illumination change, viewpoint variation, and worn or wrinkled banknotes during recognition, we propose a method that using finger pointing as HCI to point out potential targeting district which we call region of interest (ROI), thereafter, we can sharply reduce the processing time by using ROI to replace original image combining with effective ORB feature. The HCI-based framework is effective in collecting more class-specific information and robust in dealing with partial occlusion and viewpoint changes. To authenticate the robustness and generalizability of the proposed approach, we have collected a large dataset of banknotes from natural scene. The proposed algorithm improved the mean average precision from 20.3% to 61.6%. The experiments result has shown the effectiveness of our proposal both on the natural scene static dataset and the dynamic video sequence.