Growing a FLOWER: Building a Diagram Unifying Flow and ER Notation for Data Science

Abstract

An ER diagram is a fundamental visual abstraction to design a database. Modern ER notation has evolved with UML symbols to represent both entities (logical level) and relational tables (physical level). On the other hand, flow diagrams (flowcharts, process flow) remain an important mechanism to visualize the main steps of a data processing pipeline. However, in modern data science projects there is a significant fraction of data that does not come from databases or data that is exported outside the database system, being processed by Python code, without any data model whatsoever. In this paper, we present a novel diagram which is built from source code and its associated browser-based GUI for collaborating on data integration and data preprocessing, mixing diverse data sources and diverse programming languages (mainly Python and SQL). Specifically, our targets are data integration, data cleaning and data transformation, which are needed to derive data sets that can be used as input for a machine learning model. We present a couple of target applications and a preliminary GUI, which partially automates diagram creation. We show our diagram has promise understanding, extending and reusing both data preparation source code and data sets.