An optimal relationship-based partitioning of large datasets

Abstract

Modern adaptive applications utilize multiprocessor systems for efficient processing of large datasets where initial and dynamic partitioning of large datasets is necessary to obtain an optimal load balancing among processors. We applied evolutionary algorithms (Genetic Algorithm and Particle Swarm Optimization) for initial partitioning, and diffusion (DR) and cut-and-paste (CP) algorithms for dynamic partitioning. Modified versions of DR and CP algorithms are developed to improve dynamic partitioning running in NUMA multiprocessor systems. The proposed algorithms were applied on datasets describing large electricity power distribution systems and experimental results prove reductions of processor load imbalance and performance improvements. © 2010 Springer-Verlag.