You Do Not Need a bigger boat: Recommendations at Reasonable Scale in a (Mostly) serverless and open stack

Abstract

We argue that immature data pipelines are preventing a large portion of industry practitioners from leveraging the latest research on recommender systems. We propose our template data stack for machine learning at "reasonable scale", and show how many challenges are solved by embracing a serverless paradigm. Leveraging our experience, we detail how modern open source tools can provide a pipeline processing terabytes of data with minimal infrastructure work.