Proceedings of the Zoo0 IEEE International Conference on Robotics 8 Automation San Francisco, CA April 2000 Detection of Human Mistakes and Misperception for Human Perceptive Augmentation: Behavior Monitoring Using Hybrid Hidden Markov Models Mitsuichi Hiratsub*,** and H. Harry Asada** * Kawasaki Heavy Industries Co., Ltd. Akashi-shi, Hyogo, Japan ** Massachusetts Institute of Technology Cambridge, MA 02 139 e-mail : hiratu ka @ mit. edu, asada @ mit.edu Abstract A method of detecting human mistakes and misperception for assisting humans in operating complex systems is presented The method is developed in the context of operating iPASS (Integrative Physical Assists and Seamless Services) system which provides a patient diverse physical aids without changing equipment. The system can serve as a bed, a walker, a stand-up and seating assistance, as well as a wheelchair. iPASS needs special care for its operations because human mistakes and misperception might lead to serious consequences such as injury and costly repair. In order to detect human mistakes and misperception in a human motion, it is important to monitor a human motion and to understand human intention. In this paper, processes of human perception and motion are treated as stochastic processes, and they are modeled by using hybrid hidden Markov models. Finally an application of this method to stand-up assistance for iPASS is described. 1. Introduction There is an increasing need for assisting humans in operating complex systems in the home as well as in hospitals and factories. While high standards of safety are required, a sophisticated system often creates higher possibilities of human mistakes with resultant serious consequences, such as injury and costly repair. In particular, systems for home use and healthcare applications need special care in the system operation due to safety requirements and involvement of people with physical impairment, the elderly. and caregivers who are lacking in experience in operating machines. Healthcare devices, such as active beds and omni-directional wheelchairs, would not be accepted by end users unless the system is very easy to operate and safe. Moreover, since those end users are mostly people with physical impairment, elderly people, and caregivers, they are often unable to operate the system properly they tend to be inconsistent due to physical impairment, confusion, memory loss, or simple mistakes. Proper guidance and instructions as well as detection of mistakes and misperception are critical to the design of these systems. In the last decade, the new technology for human-machine interface is emerging that provides an intuitive interface to realize user-friendly systems and help humans work on the task. Wellner [ I ] developed a system that gave intuitive human-machine interfaces using projected images onto a working desk. Rauterberg et al. [2] conducted experiments to demonstrate this kind of human-machine interface to be important and helpful especially for elderly people. While these systems interacted with humans in a passive manner, Sharma et al. [3] proposed a system that gave instructions to a human in manual assembly tasks in accordance with the assembly state. This human-machine interface provided an assisting function to correct a wrong operation by the human, but it didn���t detect human confusion and misperception in the operation. Our basic assumption is that wrong actions in an operation are caused by human mistakes and misperception, and these mistakes and misperception appear as the inconsistency of a physical state of human actions with a perceived state of human actions. If human mistakes and misperception are detected through human actions, a serious accident could be avoidable beforehand. The goal of this paper is to develop an effective method for detecting human mistakes and misperception, and guiding an end user to an appropriate procedure. In this paper, a method for detecting human mistakes and misperception will be developed for assisting humans in operating complex systems. m e method will be developed in the context of operating iPASS Chtegrative _Physical Assists and Seamless Services) that is reconfigured to diverse physical aids. i.e. a bed, a walker. a stand-up and seating assistance as well as a wheelchair. Since iPASS is a multi- functional and sophisticated system, its operations are rather complex and need special caution. Mistakes and misperception may cause accidents and result in injuries for both patients and caregivers. This paper focuses on the development of human action models that would elucidate the process of fault operations, confusion. and mistakes. A hybrid model combining a hidden Markov model (HMM) and a finite automaton (FA) is used to develop a human action model. Furthermore, a method to model human actions human intention, and task process is described. Finally, an application of the method to standing-up assistance of iPASS is presented. 0-7803-5886-4/00/$1 O.OO@ 2000 IEEE 577