Orchestration

The goal of Orchestration is that the study runs reliably end-to-end, even as units are added, collaborators change, and the infrastructure underneath it evolves. The mechanism is a scientific workflow management system (SWMS), a tool that reads a description of your units and their dependencies and takes over scheduling, data movement, and recovery from interruption.

This pillar is where the other two come together: the environment each unit declares activates its isolation strategy, and its declared inputs and outputs are the boundaries where interoperation happens. Four questions identify the right orchestration strategy for your study.

Is the study complex enough for a SWMS at all? How many units are there, and how tangled are their dependencies? A study with three units that runs once is fine as a script. The threshold, and what adopting a SWMS looks like when you cross it, is Orch1.

Does the study execute across different infrastructure? Some units on a laptop, others on an HPC cluster or in the cloud? Then you need executors that speak to each scheduler, and a configuration that keeps unit code out of it. That is Orch2.

Would restarting from scratch on failure be costly? If individual units run for hours, or the study executes thousands of them, an interruption must not throw away completed work. How the two major SWMSs track and skip completed units, and where their semantics differ, is Orch3.

Does any unit need to run as a long-lived service? A model server in a Pattern C loop must stay alive while a client queries it, which breaks the start-run-exit assumption SWMSs make about units. The workarounds and their failure modes are Orch4.

Throughout this pillar the guide uses Nextflow and Snakemake as representative SWMSs, because they are widely used, free, and support conda and containers natively. The approach itself is not tied to either; the same decisions apply to any workflow manager you might already have.