Conceptual design of innovative speech interfaces with augmented reality and interactive systems for controlling loader cranes

Abstract

The paper presents a concept of implementation of augmented reality, interactive systems and an operator’s speech interface for controlling lifting devices. The aim of the experimental research is to design a prototype of an innovative system for controlling a mobile crane, equipped with a vision and sensorial system, interactive manipulators with force feedback, as well as a system for bi-directional voice communication through speech and natural language between an operator and the controlled lifting device. The system is equipped with several adaptive intelligent layers for human biometric identification, speech recognition, word recognition, analysis and recognition of commands and messages, sentence meaning analysis, command effect analysis and safety assessment, process supervision and human reaction assessment. The article presents the designed structure of an innovative system for interaction of lifting devices with their operators, which provides versatility in terms of application of the system when used for controlling and supervising modern machines and devices in conditions of difficulty or increased risk.