Detecting metachanges in data streams from the viewpoint of the MDL principle

Abstract

This paper addresses the issue of how we can detect changes of changes, which we call metachanges, in data streams. Ametachange refers to a change in patterns of when and howchanges occur, referred to as Ĝmetachanges along timeĝ and Ĝmetachanges along stateĝ, respectively. Metachanges along time mean that the intervals between change points significantly vary, whereas metachanges along statemean that themagnitude of changes varies. It is practically important to detectmetachanges because they may be early warning signals of important events. This paper introduces a novel notion of metachange statistics as a measure of the degree of a metachange. The key idea is to integrate metachanges along both time and state in terms of Ĝcode lengthĝ according to the minimum description length (MDL) principle. We develop an online metachange detection algorithm (MCD) based on the statistics to apply it to a data stream. With synthetic datasets, we demonstrated that MCD detects metachanges earlier and more accurately than existing methods. With real datasets, we demonstrated that MCD can lead to the discovery of important events that might be overlooked by conventional change detection methods.