Orch1 >>> Automation

Recommendation. Coordinate the study with a scientific workflow management system once it outgrows a script: when it must run repeatedly, when units span different environments or infrastructure, or when manual data movement becomes a bottleneck. Express every unit as a contract declaring its inputs, outputs, script, and compute environment.

When this applies

A study with few units and few collaborators that runs once is legitimately a shell script; adopting a SWMS there is overhead without benefit. The signals that the threshold has been crossed are concrete: we rerun the study with variations, we activate environments by hand in the right order, we copy files between machines to feed the next step, or a second collaborator needs to run the whole thing unaided.

Why

The unit contract is the actual core of this recommendation, more than the automation itself. When every unit declares what it consumes, what it produces, how it runs, and in which environment, three things follow. The SWMS can execute the study end-to-end, deciding order and parallelism from the declarations. The isolation strategy of each unit is activated automatically, because the environment declaration points at the conda environment or container image from the Isolation pillar. And collaborators can understand and contribute to a unit from its contract alone, without reading the implementation of every other unit, which is what makes a study survive people joining and leaving.

In practice

Nextflow and Snakemake are the representative choices, and the contract looks similar in both. In Nextflow, a unit is a process:

process trainSurrogate {
    conda 'units/surrogate/environment.yml'

    input:
    path design
    path responses

    output:
    path 'model.rds'

    script:
    """
    Rscript train.R ${design} ${responses} model.rds
    """
}

In Snakemake, a unit is a rule, and dependencies are expressed through file paths:

rule train_surrogate:
    input:
        design="results/design.csv",
        responses="results/responses.csv",
    output:
        "results/model.rds",
    conda:
        "units/surrogate/environment.yml"
    shell:
        "Rscript train.R {input.design} {input.responses} {output}"

Both declare the same four things. The conda: line can point at a lock file or be swapped for a container directive, which is how Iso1 and Iso2 plug in; the declared inputs and outputs are the boundary files whose formats Int1 governs.

For choosing between the two, the practical differences for a newcomer are these. Snakemake runs on Linux, macOS, and Windows; Nextflow runs on Linux and macOS, with Windows supported through WSL. Extending Snakemake means writing Python; extending Nextflow means Groovy on the JVM, which is a real consideration if nobody on the team has touched either. Snakemake's file-path model tends to feel natural to people who already think in terms of files on disk, while Nextflow's channel model handles complex fan-outs more directly (more on that split in Orch2). For container execution, Snakemake's native runtime is Apptainer and it converts declared Docker images automatically. Both are solid choices; the GPE benchmarking use case implements the same study in both, with the same unit contracts, precisely to show the approach does not depend on the pick.

Whichever we choose, unit code stays ignorant of the SWMS. A unit should be a program that takes input paths and produces output paths, runnable on its own with its environment activated. The contract wraps the unit; it should never reach inside it. Units then stay testable in isolation and portable to the other SWMS, or to none.

Related

Where units execute once the SWMS is in charge is Orch2; not repeating completed work is Orch3. The environment each contract declares comes from Iso1 or Iso2.