Can Data Fusion Increase the Performance of Action Detection in the Dark?

Abstract

Automated human action detection and recognition is a challenging research problem due to the complexity of its data. Contextual data provides additional cues about the actions like if we know car and man, we can short-list actions involving car and man, i.e., driving, opening the car door etc. Therefore, such data can play a pivotal role in modelling and recognizing human actions. However, the visual context during night is often badly disrupted due to clutter and adverse lighting conditions especially in outdoor environments. This situation requires the visual contextual data fusion of captured video sequences. In this paper, we have explored the significance of contextual data fusion for automated human action recognition in video sequences captured at night-time. For this purpose, we have proposed an action recognition framework based on contextual data fusion, spatio-temporal feature fusion and correlation filtering. We have performed experimentation on multi-sensor night vision video streams from infra-red (IR) and visible (VIS) sensors. Experimental results show that contextual data fusion based on the fused contextual information and its colourization significantly enhances the performance of automated action recognition.