Feature matching of simultaneous signals for multimodal synchronization

Abstract

The process of synchronizing multimodal data is used within numerous applications, including musical improvisation, interactive multimedia systems such as gaming, and exercise/sport tools. This project focuses on techniques to detect and match patterns and features from signals captured from different modalities, with particular interests in time-invariant approaches and digital warping for synchronization. Through applying feature detection algorithms and segment matching, the project aims to identify associated components within each signal for comparison with features from other associated signal streams, in order to provide feature matching. This has resulted in a framework that can be used to aid the development of a broad range of multimodal and data fusion systems. The paper also discusses a virtual conducting application as a test case that has been developed with the framework. The application alters the tempo of music playback according to physical conducting gestures. © 2013 Springer-Verlag.