Scalable data-oriented replication with flexible consistency in real-time data systems

Abstract

Scalability is an increasingly important target for distributed real-time databases. Replication is widely applied to improve the scalability and availability of data. With full replication, database systems cannot scale well, since all updates must be replicated to all nodes, whether or not they are needed there. With virtual Full Replication, all nodes have an image of a fully replicated database and the system manages the knowledge of what is needed for each node to adapt to the actual needs, so that the system can be more scalable. This work proposes a scalable and consistent replication protocol using an adaptive clustering technique that dynamically detects the new data requirements. Because time is critical in such systems, the clustering technique must take into account both the communication time cost and the timing properties of the data. The proposed protocol also proposes a new updated method for addressing the temporal inconsistency problem by skipping unnecessary operations. It allows many database nodes to update their data concurrently, without any need for distributed synchronization. It uses state-transfer propagation with on-demand integration techniques to reduce the temporal inconsistency. The experimental results show the ability of the proposed protocol to reduce the system resources consumed and improves system scalability while maintaining consistency.