Scalable distributed data analysis on structured p2p network

Abstract

In recent years, a lot of Internet of Things (IoT) devices have been developed, so we can obtain a huge amount of data (big data) from the IoT devices. In order to utilize the big data, a scalable data analysis system is required. Therefore, in this paper, I propose a scalable distributed data analysis system on a structured P2P network. In the proposed system, the IoT devices communicate with each other as nodes of a ring-type structured P2P network such as Chord. When a node requests a data analysis process, each node performs a part of the data analysis process, and the request node aggregates the partial analysis results. In my previous study, I made a scalable distributed aggregation system which calculates summations or averages of values obtained by each node. The proposed system is an extended system of the previous study, but the proposed system supports not only simple data aggregation but also data analysis such as Principal Component Analysis. In this paper, I explain how to analyze big data on a structured P2P network. In addition, I also present some simulation results, and I show that the amount of communication data required for each node is O(log N), where N is the number of nodes.