Architectural Design Criteria for Evolvable Data-Intensive Machine Learning Platforms

Abstract

Recent advances in Artificial Intelligence (AI) have fostered a widespread adoption of Machine Learning (ML) capabilities within many products and services. However, most organizations are not well suited to fully exploit the strategic advantages of AI. Implementing ML solutions is still a complex endeavor due to the fast-pace evolution and the intrinsic exploratory nature of state-of-the-art ML techniques. In many respects, the evolution of data platforms through highly parallel or high performance technologies have focused on the capacity to massively process the elements consumed by these ML models. This separate consideration renders reference architectures to be either suited for analytics consumption, or for raw storage. There is no joint consideration for the complete cycle of data management, models development, and serving with feedback and human-in-the-loop requirements. This paper introduces design criteria conceived to help organizations to architect and implement data platforms to effectively exploit their ML capabilities. The main objective of this work is to expedite the development of data platforms for ML by avoiding common implementation mistakes. The proposed guideline constitutes the methodical articulation of the empirical knowledge acquired over the last years designing, developing, evolving and maintaining a broad spectrum of relevant industry-oriented Data and AI solutions. We have focused on evaluating our proposal by assessing the functionality and usability of the architectures and implementations originated from our design criteria.