Real-Time Facial Expression Recognition and Speech Tran-scripts over an on-premise Video Conference Application

Abstract

Since Covid-19 pandemic outbreak, organizations and individuals have had to use video conference applications increasingly. However, the commercial video conference applications are expensive, and feature limited. This paper discusses how to enable organizations to host on-premise video conference applications. Then, it explores assisting organization's stakeholders with making decisions based on facial expressions of video conference attendees. Moreover, it facilitates transcribing speech into text to enable deaf persons to participate in online conferences. Technologies and tools used in addressing these challenges respectively are: (i) Web Real Time Communication (WebRTC) project, (ii) Tensorflow.js library, (iii) and Web Speech Application Programming Interface (API). This paper depends on integration between a collection of technologies , libraries, standards, and protocols. Most of them can be managed using JavaScript framework. Hence, load of the performance is distributed on each client-side device. The proposed on-premise video conference application has been enhanced through including facial expression recognition with 66% high accuracy while the speech-into-text feature with Word Error Rates (WER) are 0 and 0.12 for British English and Egyptian Arabic, respectively.