DFEP: Distributed Funding-based Edge Partitioning

Abstract

As graphs become bigger, the need to efficiently partition them becomes more pressing. Most graph partitioning algorithms subdivide the vertex set into partitions of similar size, trying to keep the number of cut edges as small as possible. An alternative approach divides the edge set, with the goal of obtaining more balanced partitions in presence of high-degree nodes, such as hubs in real world networks, that can be split between distinct partitions. We introduce DEEP, a distributed edge partitioning algorithm based on the metaphor of currency distribution. Each partition starts from a random edge and expands independently by spending currency to buy neighboring edges. After each iteration, smaller partitions receive an higher amount of currency to help them recover lost ground and reach a similar size to the other partitions. Simulation experiments show that DEEP is efficient and obtains consistently balanced partitions. Implementations on both Hadoop and Spark show the scalability of our approach.