Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is fulfilled. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by another one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an exemplary implementation of this API for Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.
翻译:科学工作流程管理系统(SWMS)和资源管理者共同确保将任务安排在提供的资源上,以便遵守所有依赖性,并实现某种优化目标,例如尽量减少差异,但在实践中,SWMS与资源管理者之间没有明确区分时间安排责任,因为没有商定将不同组成部分之间的关切分开。这有两种后果。首先,SWMS与资源管理者之间缺乏一个标准化的API以交换时间安排信息会妨碍可移植性。当一个组成部分被另一个组成部分取代时,它需要付出昂贵的调整(例如,SWMS与另一个SWMS基础,在同一资源管理者中,一个SWMS;第二,由于功能重叠,目前设施实际上往往有两个调度者,在不完整的信息之下作出部分时间安排决定,导致次优化的工作流程安排。在本文件中,我们建议SWMS与资源管理者之间建立一个简单的REST接口,让任何SWMS向资源管理者传递更清洁的工作流程信息,从而能够最充分地作出知情的时间安排决定。我们为下游的AVIS/RLA提供了堪称性的配置,在下一个版本上,而下一个版本的AVIPER将显示我们的平均战略的运行。