Hand-drawn symbol recognition in immersive virtual reality using deep extreme learning machines

Abstract

Typical character and symbol recognition systems are based on images that are drawn on paper or on a tablet with actual physical contact between the pen and the surface. In this study, we investigate the recognition of symbols that are written while the user is immersed inside a room scale virtual reality experience using a consumer grade head-mounted display and related peripherals. A novel educational simulation was developed consisting of a virtual classroom with whiteboard where users can draw symbols. A database of 30 classes of hand-drawn symbols created from test subjects using this environment is presented. The performance of the symbol recognition system was evaluated with deep extreme learning machine classifiers, with accuracy rates of 94.88% with a single classifier and 95.95% with a multiple classifier approach. Further analysis of the results obtained support the conclusion that there are a number of challenges and difficulties related to drawing in this type of environment given the unique constraints and limitations imposed by virtual reality and in particular the lack of physical contact and haptic feedback between the controller and virtual space. Addressing the issues raised for these types of interfaces opens new challenges for both human-computer interaction and symbol recognition. Finally, the approach proposed in this paper creates a new avenue of research in the field of document analysis and recognition by exploring how texts and symbols can be analyzed and automatically recognized in virtual scenes of this type.