Reusable Big Data System for Industrial Data Mining - A Case Study on Anomaly Detection in Chemical Plants

Abstract

Industrial data mining projects in general and big data mining projects in particular suffer from long project execution. The resulting high costs render many interesting use cases otherwise economically unattractive. This contribution shows on the example of anomaly detection for process plants, how the major obstacles - namely the inefficient development tools for Big Data Frameworks like Apache Hadoop and Spark and the lack of reuse of software artifacts across different projects can be overcome. This is achieved by selecting an application case that shares considerable commonalities across different projects and providing a supported project workflow implemented in a scalable and extensible big data architecture.