Data Ingestion using a Novel Method: H-Stream Framework

Abstract

Current huge volumes of data is generated from wide variety of data sources and there is lot of demand for processing, this data. Apache Hadoop is designed for batch processing. Though Hadoop is used for batch processing there is lot of requirement for real time stream processing and querying on unstructured data. Data ingestion tools of Hadoop are playing a key role in processing of streamed log data. With the increase of volume of the data performance of data ingestion tools goes down linearly. In this paper we discuss solutions for performance issues of data ingestion tools, capturing and processing of streamed multimedia data along with real-time stream processing with the help of frame work known as H-Stream framework. (Abstract)