Byte-index chunking algorithm for data deduplication system

Abstract

This paper presents an algorithm and structure for a deduplication method which can be efficiently used for eliminating identical data between files existing different machines with high rate and performing it within rapid time. The algorithm predicts identical parts between source and destination files very fast, and then assures the identical parts and transfers only those parts of blocks that proved to be unique region. The fundamental aspect of reaching faster and high scalability determining duplicate result is that data are expressed as fixed-size block chunks which are distributed to "Index-table" by chunk's both side boundary values. "Index-table" is a fixed sized table structure; chunk's boundary byte values are used as their cell row and column numbers. Experiment result shows that the proposed solution enhances data deduplication performance and reduces data storage capacity extensively. © 2013 SERSC.