A Prototype for Mexican Sign Language Recognition and Synthesis in Support of a Primary Care Physician

Abstract

Few hearing people know and use Mexican Sign Language (MSL). Consequently, this is the main barrier between people who having total or partial hearing loss and hearing people. This study proposes a system that recognizes and animates in real time a set of signs belonging to the semantic field of general medicine consultation services. Therefore, a linkage between a hearing doctor and a deaf patient can be established in a non-intrusive way and with easy dynamic interaction. Our main contribution is a bidirectional translator system for Mexican Sign Language in the context of primary care health services, in addition to basic signs to fingerspell alphabet and numbers as a complement to provide personal information such as name, age, etc. The recognition module uses a Microsoft Kinect sensor to obtain sign trajectories and images to feed hidden Markov Models (HMMs) for processing sign samples in real time. The experiments showed the recognition of 82 different signs by 22 participants. As a result, accuracy and F1 scores average rates of 99% and 88%, respectively, were obtained.