Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions

The increasing amounts of data related to the execution of scientific workflows has raised awareness
of their shift towards parallel data-intensive problems. In this paper, we deliver our experience combining
the traditional high-performance computing and grid-based approaches with Big Data analytics
paradigms, in the context of scientific ensemble workflows. Our goal was to assess and discuss the
suitability of such data-oriented mechanisms for production-ready workflows, especially in terms of
scalability. We focused on two key elements in the Big Data ecosystem: the data-centric programming
model, and the underlying infrastructure that integrates storage and computation in each node. We
experimented with a representative MPI-based iterative workflow from the hydrology domain, EnKFHGS,
which we re-implemented using the Spark data analysis framework. We conducted experiments on
a local cluster, a private cloud running OpenNebula, and the Amazon Elastic Compute Cloud (AmazonEC2).
The results we obtained were analysed to synthesize the lessons we learned from this experience, while
discussing promising directions for further research.