Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges

Abstract

In many applications (e.g., anomaly detection and security systems) of smart cities, rare events dominate the importance of the total information on big data collected by the Internet of Things (IoT). That is, it is pretty crucial to explore the valuable information associated with the rare events involved in minority subsets of the voluminous amounts of data. To do so, how to effectively measure the information with the importance of the small probability events from the perspective of information theory is a fundamental question. This paper first makes a survey of some theories and models with respect to importance measures and investigates the relationship between subjective or semantic importance and rare events in big data. Moreover, some applications for message processing and data analysis are discussed in the viewpoint of information measures. In addition, based on rare events detection, some open challenges related to information measures, such as smart cities, autonomous driving, and anomaly detection in the IoT, are introduced which can be considered as future research directions.