Data stream analytics as cloud service for mobile applications

Abstract

Many mobile applications are based on cloud services such as location service, messaging service, etc. Currently most cloud services are based on statically prepared information rather than the real-time analytics results of dynamically captured events. A paradigm shift is to take Continuous Stream Analytics (CSA) as a cloud service, which, however, poses several specific challenges in scalability, latency, time-window semantics and transaction control. In this work we extend the SQL query engine to unify the processing of static relations and dynamic streams for providing the platform support of CSA service. This platform is significantly differentiated from the current generation of stream processing systems which are in general built separately from the database engine thus unable to take advantage of the functionalities already offered by the existing data management technology, and suffer from the overhead of inter-platform data access and movement. To capture the window semantics in CSA, we introduce the cycle-based query model and support it in terms of the cut-and-rewind query execution mechanism. This mechanism allows a SQL query to run cycle by cycle for processing the unbounded stream data chunk by chunk, but without shutting the query instance down between chunks for continuously maintaining the application state across the execution cycles, as required by sliding-window oriented operations. We also propose the cycle-based transaction model with cycle-based isolation and visibility. To scale-up analytics computation, we introduce the parallel infrastructure with multi-engines cooperated and synchronized based the common data chunking criteria without centralized coordination. To scale-up service provisioning, we investigate how to stage the continuously generated analytics results efficiently through metadata manipulation without physical data moving and copying. We have prototyped our approach by extending the PostgreSQL, resulting in a new kind of tightly integrated, highly efficient platform for providing CSA service. We tested the throughput and latency of this service using a well-known stream processing benchmark and with WebOS based Palm phones. The test results show that the proposed approach is highly competitive. Providing CSA cloud service using HP Neoview parallel database engine is currently explored. © 2010 Springer-Verlag.