Automatic web-based question answer generation system for online feedable new-born chatbot

Abstract

The knowledge bases of Chatbots are built manually, which is difficult and time consuming to create and maintain. The idea of automatically building a Chatbot knowledge base from the web has emerged in recent years. Question Answer (QA) pairs are acquired from existing online forums. Little work has been done on generating questions from existing fact or fictional sentences. Two main contributions are presented in this paper. The first contribution is generating factual questions from sentences gathered by a web spider; the raw text sentences are extracted from the HTML and pre-processed. Named Entity (proper none) Recognition (NER) is used in addition to verb tense recognition in order to identify the factual sentence category. Specific rules are built to categorize the sentences and then to generate questions based upon them. The second contribution is to generate a new born Chatbot database by placing the resultant QA pairs into an SQLite database built for this purpose. The new built database is used to nurture a Chatbot that can simulate the personality of a desired figure or behavior of an object. The footballer David Beckham is used as an example and the data used is acquired from a page on about him on Wikipedia. The resulting QA pairs are presented and a subjective assessment shows considerable enhancement in QA pairs’ generation over a comparative system.