Discovering functional dependencies in vertically distributed big data

Abstract

The issue of discovering FDs has received a great deal of attention in the database research community. However, as the problem is exponential in the number of attributes, existing approaches can only be applied on small centralized datasets. It is challenging to discover FDs from big data, especially if data is distributed. We present a new algorithm DFDD for discovering all functional dependencies in parallel in vertically distributed big data following a breadth-first traversal strategy of the attribute lattice that combines efficient pruning. We verify experimentally that our approach can process distributed big datasets and it is scalable with the number of cluster nodes and the size of datasets