Résultat de la recherche
Applying big data paradigms to a large scale scientific workflow: Lessons learned and future directions
The increasing amounts of data related to the execution of scientific workflows has raised awareness of their shift towards parallel data-intensive problems. In this paper, we deliver our experience combining the traditional high-performance computing and grid-based approaches with Big Data analytics paradigms, in the context of scientific ensemble workflows. Our goal was to assess and discuss the suitability of such data-oriented mechanisms for production-ready workflows, especially in terms of scalability. We focused on two key elements in the Big Data ecosystem: the data-centric programming model, and the underlying infrastructure that integrates storage and computation in each node. We experimented with a representative MPI-based iterative workflow from the hydrology domain, EnKFHGS, which we re-implemented using the Spark data analysis framework. We conducted experiments on a local cluster, a private cloud running OpenNebula, and the Amazon Elastic Compute Cloud (AmazonEC2). The results we obtained were analysed to synthesize the lessons we learned from this experience, while discussing promising directions for further research.
Wireless Mesh Networks and Cloud Computing for Real Time Environmental Simulations
Predicting the influence of drinking water pumping on stream and groundwater levels is essential for sustainable water management. Given the highly dynamic nature of such systems any quantitative analysis must be based on robust and reliable modeling and simulation approaches. The paper presents a wireless mesh-network framework for environmental real time monitoring integrated with a cloud computing environment to execute the hydrogeological simulation model. The simulation results can then be used to sustainably control the pumping stations. The use case of the Emmental catchment and pumping location illustrates the feasibility and effectiveness of our approach even in harsh environmental conditions.
Integrating hydrological modelling, data assimilation and cloud computing for real-time management of water resources
Online data acquisition, data assimilation and integrated hydrological modelling have become more and more important in hydrological science. In this study, we explore cloud computing for integrating field data acquisition and stochastic, physically-based hydrological modelling in a data assimilation and optimisation framework as a service to water resources management. For this purpose, we developed an ensemble Kalman filter-based data assimilation system for the fully-coupled, physically-based hydrological model HydroGeoSphere, which is able to run in a cloud computing environment. A synthetic data assimilation experiment based on the widely used tilted V-catchment problem showed that the computational overhead for the application of the data assimilation platform in a cloud computing environment is minimal, which makes it well-suited for practical water management problems. Advantages of the cloud-based implementation comprise the independence from computational infrastructure and the straightforward integration of cloud-based observation databases with the modelling and data assimilation platform.
Real-time Environmental Monitoring for Cloud-based Hydrogeological Modeling with HydroGeoSphere
This paper describes an architecture for real-time environmental modeling. It consists of a wireless mesh network equipped with sensors and a cloud-based infrastructure to perform real-time environmental sim- ulations using a physics-based model combined with an Ensemble Kalman Filter. The purpose of the system is to optimize groundwater abstraction close to a river. These initial studies demonstrate that the cloud infrastructure can simultaneously compute a large number of simula- tions, thus allowing for the implementation of Ensemble Kalman Filters in real-time.
Approaches for Cloudification of Complex High Performance Simulation Systems
Scientific computing is often associated with ever-increasing need for computer resources to conduct experiments, simulations and gain outcomes in a reasonable time frame. While local infrastructures could hold substantial computing power and capabilities, researchers may still reach the limit of available resources. With continuously increasing need for higher computing power, one of the solutions could be to offload certain resource-intensive applications to a cloud environment with resources available on-demand. In this thesis, we will address the problem of migrating MPI-based scientific applications to clouds. Specifically, we will concentrate on scientific simulators, which implement the iterative Monte Carlo method. To tackle the identified problem, we will (a) overview high performance and cloud computing domains, (b) analyze existing simulation problem types, (c) introduce an example Monte Carlo simulator, (d) present two cloudification methodologies, (e) apply the methodologies to the example simulator, and (f) evaluate the potential application of methodologies in a real case study.