A new massive data processing framework under cloud environment for digital community

Abstract

Digital community involves e-government, e-commerce, smart health and other applications. With the increase of customers and types of business, it becomes more important for digital community to process massive data efficiently. Although, the current cloud-based applications can provide some elastic and on-demand calculation abilities to digital community, their underlying programming models still have certain limitations. This study aims to provide a new framework of massive data processing for digital community. In the framework, multiple programming models are adopted and each programming model handles the specific calculations that they do best. These calculations mainly include embarrassingly parallel calculation, iteration calculation and data-dependent complex calculation. To improve the performance of the framework, the programming model connection pool and the virtual subnet are designed and applied. Compared to Hadoop and its modified version, on average, the proposed framework can reduce runtime by 1.32. The experimental results show that the proposed framework has higher generality and efficiency. Moreover, it is reasonable and valuable for digital community to analyze comprehensively trade area on geographical location and business volume. © 2013 Asian Network for Scientific Information.