Due to lack of software for conducting data farming experiments, we developed a new system from scratch, called Scalarm, which stands for Massively Scalable Platform for Data Farming. Scalarm intends to fulfill the following requirements:
- ● support all phases of a data farming experiment, starting from the experiment design phase, through simulation execution and progress monitoring, to statistical analysis of results,
- ● support different sizes of experiments from dozens to millions of simulations through massive scalability,
- ● support for heterogeneous computational infrastructure including private servers, computer clusters, grids and clouds.
Scalarm’s architecture utilizes a service-oriented approach with an additional modification, which addresses the scalability requirement. To cope with the requirement, we do not operate on the level of components and services, which represent single instances only. Instead, we have to extend the meaning of an application's modularization unit to embrace the scalability feature. Thus, each Scalarm service can consist internally of a number of component’s instances, which provides exposed functionality, and a load balancer, which constitutes a single entry point to the service. An overview of the architecture is depicted in Fig. 1.
The Scalarm platform includes the following services:
- ● Experiment manager, which handles all interaction with users through a Graphical User Interface. On the one hand, it constitutes a gateway to the platform for analysts, i.e. provides a coherent view of information about all running and completed data farming experiments, and enables analysts to create new experiments or to conduct statistical analysis on existing ones. On the other hand, experiment manager is responsible for scheduling simulations to simulation managers.
- ● Storage manager is an equivalent of the persistence layer concept but in a form of a separate service. Other services use this storage manager to store different types of data: structural information about each executed simulation and experiment, and actual results of simulations, which may be either binary or text data.
- ● Simulation manager is an intelligent wrapper for actual simulations, which can be deployed on different e-infrastructures. It can be treated as an implementation of the Pilot job concept, i.e. a special application that intends to acquire computational resources to run actual applications. However, while the Pilot job concept was created for Grid environments only, simulation manager is infrastructure independent.
- ● Information service is an implementation of the service locator pattern, known from SOA-based systems. It is a "well-known" place for each component in the system, which stores information about other components' locations.
- ● Data explorer encapsulates data analysis algorithms built-in the Scalarm platform.
- ● Pathfinder is the latest Scalarm service and it provides advanced features related to automatic input parameter space exploration. The user with a few clicks can start a new experiment where input parameter space will be determined at runtime using selected algorithms, e.g. global optimization methods such as simulated annealing or sensitivity analysis algorithms lik Morris or Sobol'.
When considering a massively self-scalable platform constituted by several loosely coupled services, the most challenging requirement is the management of multiple instances of each service running in parallel, in such a way that all the instances are treated as a single unit by external clients. To cope with the requirement, we can’t operate on the level of services treated as single instances only. Instead, we have to extend the meaning of a service as a software modularization unit to embrace the scalability feature. Thus, we introduce the concept of a self-scalable service, whose structure is depicted in Fig. 2, which can be defined as a group of service instances with the same functionality, which can be accessible as a single unit by external clients.
To demonstrate how Scalarm works in practice, we prepared a quick start demonstration. It shows how you can install Scalarm on your own machine and start using it with an example application.
We also have a virtual machine with Scalarm and Pegasus workflow management system installed. A virtual machine image (compatible with virtual box) is available here. And here is a slide deck with Scalarm tutorial: