Clustering analysis to evaluate usability of work-flow systems and to monitor proficiency of workers

Abstract

In order to evaluate usability, we often interview the users with using questionnaire sheets. However, conventional processing methods for questionnaire data are so simple that we cannot mine maximum information from the users' opinions. This paper proposes a new method for deeper analysis of the user opinions. Using the vector quantization method, we can classify users into groups reflecting their skill grade. Also, by observing learning curves of the tasks, we can evaluate hardness of mastering each task and detect the defects of the work-flow to be improved. This paper explains the idea and mechanics of the method with referring an actual example, which is a questionnaire investigation for workers in a real office to examine usability problems on their works. The result of data analysis pointed out several very hard defects in the work-flow system, on which not only the novices but the experts are also facing difficulties. Those very hard defects cannot be solved by experience or training, because the experts cannot cope with them. Thanks to the vector quantization analysis, we can distinguish between difficult points that can be solved by workers' experience and such very hard defects that require drastic reforms for improvement from. © 2011 Springer-Verlag.