ISDI: A New Window-Based Framework for Integrating IoT Streaming Data from Multiple Sources

Abstract

Due to the rapid advancement in Internet of Things (IoT), myriad systems generate data of massive volume, variety and velocity which traditional databases are unable to manage effectively. Many organizations need to deal with these massive datasets that encounter different types of data (e.g., IoT streaming data, static data) in different formats coming from multiple sources. Different data integration mechanisms are designed to process mostly static data. Unfortunately, these techniques are not adequate to integrate IoT streaming data from multiple sources. In this paper, we identify the challenges of IoT streaming data integration (ISDI). A generic window-based ISDI approach is proposed to deal with IoT data in different formats and subsequently introduced the algorithms to integrate IoT streaming data obtained from multiple sources. In particular, we extend the basic windowing algorithm for real-time data integration and to deal with the timing alignment issue. We also introduce a de-duplication algorithm to deal with data redundancies and to demonstrate the useful fragments of the integrated data. We conduct several sets of experiments and quantify the performance of our proposed window-based ISDI approach. The experimental results, performed on several IoT datasets, show the efficiency of our proposed ISDI solution in terms of processing time.