Rare event-prediction with a hybrid algorithm under power-law assumption

Abstract

We present an algorithm for predicting both common and rare events. Statistics show that occurrences of rare events are usually associated with common events. Therefore, we argue that predicting common events correctly is an important step toward correctly predicting rare events. The new algorithm assumes that frequencies of events exhibit a power-law distribution. The algorithm consists of components for detecting rare event types and common event types, while minimizing computational overhead. For experiments, we attempt to predict various fault types that can occur in distributed systems. The simulation study driven by the system failure data collected at the Pacific Northwest National Laboratory (PNNL) shows that fault-mitigation based on the new prediction mechanism provides 15% better system availability than the existing prediction methods. Furthermore, it allows only 10% of all possible system loss caused by rare faults in the simulation data.