An ICT Project Case Study from Education: A Technology Review for a Data Engineering Pipeline

Abstract

The paper presents a brief technology survey of existing tools to implement data ingestion pipelines in a classical Data Science project. Given the emergent nature of technologies and the challenges associated with any Big Data project, we propose to identify and discuss the main components of a data pipeline, from a data engineering perspective. The data pipeline is showcased with a case study from an ICT university project, where several teams of master students competed towards designing and implementing the best solution for a manufacturing data pipeline. The project proposes a research-based multidisciplinary approach to education, aiming at empowering students with a novel role in the process of learning, that of knowledge creators. Therefore, on the one hand, the paper discusses the main components of a Big Data pipeline and on the other hand it shows how these components are addressed and implemented within a concrete ICT project from education, realized in tight relation with the IT industry.