A tool for supporting progressive refinement of wizard-of-Oz experiments in natural language

Abstract

Wizard-of-Oz techniques are an important method for collecting data about the behavior of students in tutorial dialogues with computers, especially when the interaction is done in natural language. Carrying out such experiments requires dedicated tools, but the existing ones have some serious limitations for supporting the development of systems with ambitious natural language capabilities. In order to better meet such demands, we have developed DiaWoZ, a tool that enables the design and execution of Wizard-of-Oz experiments to collect data from dialogues and to evaluate components of dialogue systems. Its architecture is highly modular and allows for the progressive refinement of the experiments by both designing increasingly sophisticated dialogues and successively replacing simulated components by actual implementations. A first series of experiments carried out with DiaWoZ has confirmed the need for elaborate dialogue models and the incorporation of implemented components for subsequent experiments. © Springer-Verlag 2004.