Monitoring High Throughput Distributed System using Statistical Data Analysis

Abstract

Monitoring high throughput distributed system by using a statistical analysis of the “historical time series” of an Instrumentation Data”. “The Pipeline has been made to process the information which can be otherwise called data pipeline, is a lot of information handling components associated in arrangement, where yield of one component is the contribution of the next one”. Several codes are giving different visualization for statistical analysis of data. “Network and Cloud Data Centers” generate a lot of data every second; this data can be gathered as period arrangement information. A time-series is a grouping taken at progressive similarly dispersed focuses in time that implies at a particular time interval to a particular time, the estimations of explicit information that was taken is known as information of a time-series. “This time-series information can be gathered utilizing framework measurements like CPU, Memory, and Disk utilization”. The TICK and ELK Stack is abbreviation for a foundation of open source instruments worked “to make collection, storage, graphing, and alerting” on time arrangement data incredibly easy. As an information collector, using Telegraf, “for storing and analyzing” information and the time-series database InfluxDB and Elasticsearch. For plotting and visualizing used Grafana and Kibana. Watchman is utilized for alert refinement and once system metrics usage exceeds the specified threshold, the alert is generated and sends it to the Telegram.