How natural is a natural interface? An evaluation procedure based on action breakdowns

Abstract

This paper describes an issue-based method to evaluate the naturalness of an interface. The method consists of the execution of a series of tasks on that interface, which is subsequently systematically analyzed to identify breakdowns in the users' actions. The systematic analysis of breakdowns is allowed by the support of video-coding software (The Observer by Noldus). This method is described on its theoretical bases and then applied to the evaluation of a natural interface, a walk-in-place locomotion system for virtual spaces called Superfeet. The procedure is comparative, since Superfeet is compared to two locomotion devices, Superfeet enhanced with headtracker and a more traditional Joypad. The test involves 36 participants (mean age = 23.68, SD = 3.14). The outcomes of the breakdown analysis are illustrated at a progressively finer level of granularity from the amount and length of breakdowns, to the circumstances of the breakdowns, to the type of actions involved in the breakdowns. The potential of this procedure for usability studies is finally synthesized. © 2011 Springer-Verlag London Limited.