A wearable multisensory, multiagent approach for detection and mitigation of acute cognitive strain: Phase I - Vocalization analysis

Abstract

While operators performing tasks with high workload can increase task performance in response to limited increases in cognitive stress, chronic or rapidly accelerating stress can exceed the operator’s ability to compensate, generating acute cognitive strain (ACS). ACS represents a state wherein performance, situation awareness and cooperativity deteriorate markedly, leading to critical errors, mishaps or casualties. Nearly two decades of augmented cognition (AugCog) research has demonstrated the utility of psychophysiologic sensing and analysis for identification and tracking of changes in cognitive state and to modulate human machine interactions for improving system task performance. The proposed approach leveraged prior efforts to modulate cognitive stress using a multiagent approach to acquire and analyze multiple Psychophysiologic sensory channels, including changes in vocalizations, to create a reliable and non-intrusive Detector of Acute Cognitive Strain (DACS). The DACS system provides an integrated wearable multi-modal Research Sensor Suite (RSS) using the open-source Adaptive Multiagent Integration (AMI) architecture, that includes analysis agents for electroencephalograph (EEG), electromyography (EMG), video oculography (VOG), vocalization, and others to identify and correlate physiological signatures with cognitive stress and strain. An online AMI agent-based processing algorithm was developed and applied to audio communications to evaluate for changes in speaker vocalization fundamental frequency (F0) and cadence (utterances per minute). This paper describes initial phase results of aerospace mishap vocalization stress marker detection, a potential element of the proposed DACS system. DACS could use these markers to trigger adaptive automation agents that reduce task load and allow pilots to prevent or recover from ACS episodes.