Description
RYAX is a workflow management system for data analytics. It enables workflow composition and orchestration upon hybrid distributed infrastructures such as HPC and Cloud. It is built on top of Kubernetes inheriting its interoperability, flexibility, fault-tolerance and powerful declarative configuration. RYAX proposes a new DSL language which facilitates the definition of workflows. These architectural choices make RYAX a perfect candidate for stream processing, even though batch processing is possible, and well adapted for deploying on hybrid distributed environments. Task placement is delegated on the Kubernetes scheduler whose modularity allows us to develop different types of scheduling algorithms. RYAX brings the convergence of HPC/AI since it enables the execution of hybrid workflows through integration/communication with the dedicated HPC or Big Data resource managers. Kubernetes currently supports different types of containers platforms (Singularity, Docker) and this is inherited by RYAX.
Integration
RYAX makes use of Kubernetes to deploy workflow tasks on the right resources. When deployed on the HPC center, Kubernetes will play the role of a meta-scheduler and will communicate with the dedicated HPC resource manager to schedule workflow tasks on the HPC cluster. In the context of HPC/HPDA convergence, Singularity containerization platform will be leveraged along with the needed open-source tools, wlm-operator and Singularity-CRI, to enable interactions between Kubernetes and the HPC resource manager. RYAX will be adapted to tightly integrate Singularity and these tools to deploy workflows using REGALE’s specialized HPC resource managers (OAR and SLURM) and Big Data/AI frameworks (such as Spark).
Sophistication
In the context of REGALE, RYAX will be extended to bring elasticity in a traditionally rigid HPC environment by integrating the BeBiDa tool and providing adaptations to handle the optimizations for improved guarantees in terms of elasticity and autoscaling. New multi-criteria scheduling algorithms combined with observability mechanisms of HPC resource managers will be implemented on top of Kubernetes and can be used by RYAX to enable efficient environment.
Please visit https://docs.ryax.tech/index.html for more information on RYAX.