WDFed: Exploiting cloud databases using metadata and RESTful APIs

Abstract

As a result of the development of Big Data and cloud databases, a huge amount of data are available on the Web, not only as dump files but also in databases. Due to the volume and heterogeneity of these data, it is a challenging task to find and consume them. To reduce the barrier of data sharing and reuse on the Web, we propose a data cataloguing framework called WDFed that combines the strength of both Linked Data and REST. WDFed adopts the Data Catalog Vocabulary (DCAT) to harmonise database metadata, and develops a two-way mapping between DCAT and a RESTful API. This framework provides an interoperable middle layer that enables humans as well as applications to discover and consume Big Data on the Web in a semi-automatic manner. The framework is implemented in a data portal called Web Observatory and we present several use cases to evaluate the framework.