Comparing Physical and Immersive VR Prototypes for Evaluation of an Industrial System User Interface

Abstract

Since immersive VR devices have become commodities, immersive environments appear as a new tool in the development of high-fidelity prototypes of systems in which the user interaction relies on expensive or unusual hardware, e.g., industrial systems. However, there is not enough evidence that the interface of a complex system and its VR counterpart have equal usability and user experience qualities. Our main objective is to assess the feasibility of carrying out studies on user-based evaluation in industrial interactive systems through immersive VR simulation. To achieve this, we compared user assessment with a conventional prototype of an industrial system with its immersive VR simulation. We performed within-subjects user testing in both the physical and the VR setups, and collected (i) experimenters’ observations on usability issues and (ii) subjective and objective measures of 16 participants. Subjective measures were taken using standardized questionnaires and objective measures by logging the elapsed time to fulfill task scenarios. Our results indicate that the perceived quality of the immersive VR system is indistinguishable from the physical counterpart regarding User Experience, usability, and cybersickness. On the other hand, the users’ performance on VR simulation was significantly slower in immersive VR. Finally, the same usability issues could be detected with either of the conditions.