Secure hadoop with encrypted HDFS

Abstract

As Hadoop becomes a popular distributed programming framework for processing large data on its distributed file system (HDFS), demands for secure computing and file storage grow quickly. However, the current Hadoop does not support encryption of storing HDFS blocks, which is a fundamental solution for secure Hadoop. Therefore, we propose a secure Hadoop architecture by adding encryption and decryption functions in HDFS. We have implemented secure HDFS by adding the AES encrypt/decrypt class to CompressionCodec in Hadoop. From experiments with a small Hadoop testbed, we have shown that the representative MapReduce job on encrypted HDFS generates affordable computation overhead less than 7%. © 2013 Springer-Verlag.