SHOWME.how

Appearance

Orch2 >>> Scheduling ​

Recommendation. Let the workflow management system decide where and how units execute: it resolves the dependency graph before execution and submits each unit to the appropriate executor, whether a local process, a SLURM job, or a cloud backend. Keep placement entirely in configuration, so moving the study to different infrastructure changes no unit code.

When this applies ​

From the moment a SWMS coordinates your study (Orch1), and especially when the study runs on more than a laptop: submitting to an HPC scheduler, running units in parallel within a resource budget, or spanning infrastructures in one run.

Why ​

Manual scheduling is the hidden cost of script-driven studies: knowing which units can run in parallel, writing SLURM submission scripts, watching queues, and starting the next stage when the previous one finishes. A SWMS derives all of that from the unit contracts. And because placement lives in configuration rather than in unit code, the same study runs on a laptop today and a cluster tomorrow, which is most of what infrastructure portability means in practice.

In practice ​

Both Nextflow and Snakemake plan statically: they resolve the full dependency graph from the unit contracts before execution begins, then submit each unit as soon as its declared inputs exist. What differs is how we express the graph. Nextflow follows a dataflow paradigm, where processes are connected by explicit channels through which outputs stream to inputs. Snakemake follows a file-based paradigm, where we name the file we want and Snakemake works backwards through matching input and output paths to determine what must run. The dataflow style handles complex fan-out and fan-in more directly; the file-based style is the shorter path when your study already thinks in files on disk.

Executors are where placement happens. In Nextflow, they are configuration profiles:

groovy
// nextflow.config
profiles {
    standard {
        process.executor = 'local'
    }
    cluster {
        process.executor = 'slurm'
        process.queue    = 'batch'
        process.time     = '4h'
        process.memory   = '8 GB'
    }
}

Running nextflow run main.nf -profile cluster submits every unit as a SLURM job; dropping the flag runs the same study locally. Nothing in any unit changes. In Snakemake (version 8 and later), executors are plugins selected on the command line:

bash
snakemake --executor slurm --jobs 50 \
  --default-resources slurm_partition=batch mem_mb=8000 runtime=240

Per-unit resource needs (cores, memory, GPU, wall time) belong in the contract too, declared per process or rule, so the executor can request the right allocation for each unit rather than one oversized allocation for everything. Both tools also cap concurrency for you: the sweep of a Pattern A study can declare a thousand tasks while the configuration limits how many run at once, locally by available cores, on a cluster by submitted jobs.

The portability claim has one honest boundary: it holds for where units run, not automatically for what they run on. A unit that needs a GPU still needs a node that has one, and a containerised unit needs the cluster-side runtime from Iso3. The hemolysis use case is the pattern in action: the study moved from laptops to an HPC cluster by switching the executor profile and building Apptainer images, with no changes to unit code.

The contracts that scheduling is derived from are Orch1. What happens when a scheduled run is interrupted halfway is Orch3. Units that must outlive a single scheduled execution are the subject of Orch4.