N-Gram and Full-Text Search Algorithm Testing for Pattern Recognition in a Chatbot Engine †

Abstract

The development of chatbots to access database services and information systems has triggered a lot of research on frameworks for service development, including the development of ISONER (Information System On Internet Messenger). This framework consists of multiple phases including pattern recognition, query processing, and response generation. In its implementation, the framework develops pattern recognition services that are currently based on Natural Language Processing (NLP). Improved pattern recognition algorithms enhance the system’s ability to accurately interpret user intent. The pattern recognition used in this research utilizes built-in plugins from MySQL, namely N-gram and Full-Text Search, which can be run directly on the MySQL engine to reduce latency and do not require another programming language. The FTS and fourgram algorithms gave the best results when applied on 100 test data points, with a threshold of 0.91, accuracy of 91%, precision of 99%, and recall of 92%; the average computation time was 19 s for 100 test data points and 2 min 49 s for 1000 data points tested simultaneously.