Out-of-Scope Intent Detection on A Knowledge-Based Chatbot

Abstract

Knowledge-based chatbot (KBC) has grown in popularity in recent years and has been widely used for various use cases. Building KBC from scratch using deep learning (DL) is challenging since no prior historical data exists. Meanwhile, DL systems need a vast Volume of data to be trained. This paper proposes a novel framework to create an intent classifier of the KBC used to detect in-scope (IS) and out-of-scope (OOS) intents. We introduce an automated queries generator to create IS intents employed as the training data from an ontology input. We utilize Bidirectional Encode Representations from Transformers (BERT) fine-tuning as the backbone of our DL system. Moreover, we present a Bayesian approach as an extension of the BERT to classify OOS queries with minimal OOS training data. The experiments result show that the proposed method manages to achieve an F1 score of 100% for IS intents and 86% for OOS queries.