An interoperability framework and distributed platform for fast data applications

Abstract

Big data developments have been centred mainly on the volume dimension of data, with frameworks such as Hadoop and Spark, capable of processing very large data sets in parallel. This chapter focuses on the less researched dimensions of velocity and variety, which are characteristics of fast data applications. The chapter proposes a general-purpose distributed platform to host and interconnect fast data applications, namely, those involving interacting resources in a heterogeneous environment such as the Internet of Things. The solutions depart from conventional technologies (such as XML, Web services or RESTful applications), by using a resource-based meta model that is a partial interoperability mechanism based on the compliance and conformance, service-based distributed programming language, binary message serialization format and architecture for a distributed platform. This platform is suitable for both complex (Web-level) and simple (device-level) applications. On the variety dimension, the goal is to reduce design-time requirements for interoperability by using structural data matching instead of sharing schemas or media types. In this approach, independently developed applications can still interact. On the velocity dimension, a binary serialization format and a simple message-level protocol, coupled with a cache to hold frequent type mappings, enable efficient interaction without compromising the flexibility required by unstructured data.