Evaluation of the Slovak spoken dialogue system based on ITU-T

Abstract

The development of the Slovak spoken dialogue system started in year 2006. The developed system is publicly available as a trial system and provides weather forecast and timetables information services. One of the very important questions is how to evaluate quality of such a system. A new method for quality assessment of the spoken dialogue system is proposed. The designed method is based on ITU-T P.851 recommendation. Three types of questionnaires were prepared - A, B and C. The questionnaires serve for obtaining information about user's background, completed interactions with the system and about overall impression of the system and its services. Scenarios, methodology of coding, classifying and rating of collected data were also developed. There are also six classes of quality for representation of system's features. Introduced method was used for evaluation of the dialogue system and timetables information service. This paper also summarizes the results of performed experiment. © 2008 Springer-Verlag Berlin Heidelberg.