An overview on data deduplication techniques

Abstract

The massive data puts forward higher requirements on the capacity of storage devices, but from a practical point of view, the increasement of capacity is far more behind the growth of data. Deduplication technique, for its high efficiency, few resource consumption and extensive application scope, comes to the fore among various data reduction techniques. The so-called data deduplication refers to find and eliminate redundant data among the storage system. For local storage system, the only one data object is needed to store to save limited storage space; for network system, not only storage space can be saved, but also transmission bandwidth can be reduced to increase the transmission rate. It is a compromise to achieve the purpose of efficient storage at cost of computational overhead. This article will introduce data deduplication techniques, describe basic principles and processes, summarize the main technique of the current study and provide recommendations for future development.