Abstract
Squire Kilometre Array (SKA) project generates almost the hugest data volume in the world. SKA data flow pipelines need almost real-time processing ability, which brings huge challenges to the execution frameworks (EF for short). We propose a cost model for a typical SKA data flow pipeline named as MID1 ICAL pipeline on Spark. By simulating the I/O of MID1 ICAL pipeline with a reduced SKA data, we evaluate several different implementations of MID1 ICAL pipeline and conclude the optimized method for this pipeline on Spark.
Cite
CITATION STYLE
Li, Z., Li, Q., Liu, Y., Wang, W., Qi, F., Chi, M., & Wang, Y. (2018). Modeling and evaluating MID1 ICAL pipeline on spark. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 10828 LNCS, pp. 825–828). Springer Verlag. https://doi.org/10.1007/978-3-319-91458-9_57
Register to see more suggestions
Mendeley helps you to discover research relevant for your work.
