Erasure-coded byzantine storage with separate metadata

Abstract

Although many distributed storage protocols have been introduced, a solution that combines the strongest properties in terms of availability, consistency, fault-tolerance, storage complexity, and concurrency has been elusive so far. Combining these properties is difficult, especially if the resulting solution is required to be efficient and incur low cost. We present AWE, the first erasure-coded distributed implementation of a multi-writer multi-reader read/write register object that is, at the same time: (1) asynchronous, (2) wait-free, (3) atomic, (4) amnesic, (i.e., nodes store a bounded number of values), and (5) Byzantine fault-tolerant (BFT), using the optimal number of nodes. AWE maintains metadata separately from bulk data, which is encoded into fragments with a k-out-of-n erasure code and stored on dedicated data nodes that support only simple reads and writes. Furthermore, AWE is the first BFT storage protocol that uses only n = 2t + k data nodes to tolerate t Byzantine faults, for any k ≥ 1. Metadata, on the other hand, is stored using an atomic snapshot object, which may be realized from 3t + 1 metadata nodes for tolerating t Byzantine faults. AWE is efficient and uses only lightweight cryptographic hash functions.Moreover, we show that hash functions are needed by any BFT distributed storage protocol that stores the bulk data on 3t or fewer data nodes.