Multisensory information processing for enhanced human-machine symbiosis

Abstract

Multisensory information processing is a basic feature of neural systems and has been exploited to facilitate development of Army systems that augment Soldier performance through multisensory displays. However, the full potential of these systems has yet to be determined and will require understanding fundamental features of the underlying neurophysiology of multisensory processing, the neuroergonomics of multisensory machine interface and analytical methods for neural signal analysis, dimensionality reduction and pattern recognition. Here, findings from basic and applied research efforts will be presented that have focused on various aspects of human (brain)-computer interfaces to uncover understanding in these areas and mediate recent technological developments in multisensory display technology, passive mental state detection, attention/orientation detection, and human activities recognition from video in general. Based on the knowledge of multisensory processes acquired from these efforts there are emerging opportunities for creating new human gesture-controlled recognition systems based upon multimodal data analysis which will allow for unprecedented human-machine symbiosis.