Orch3 >>> Incremental Execution

Recommendation. Rely on the workflow management system to track which units have completed successfully and to skip them on rerun, so a failure mid-run restarts from the last completed unit rather than from scratch. Learn the exact resumption semantics of the tool in use; Snakemake and Nextflow differ in ways that cost real compute time when misunderstood.

When this applies

Whenever restarting from scratch would be expensive: individual units that run for hours, or studies that execute a large number of units, as any serious Pattern A sweep does. Interruptions are not exceptional on shared infrastructure; wall-time limits, node failures, and a bug in the last unit of the chain all end runs halfway.

Why

Without incremental execution, an interruption means manually working out which steps completed and rerunning the remainder by hand, which is fragile and slow. With it, resumption is one command. The SWMS can do this because the unit contracts tell it exactly what each unit consumes and produces, so it can prove that a completed unit's work is still valid. This is also the first place collaborators get burned by tool differences, because the two major SWMSs resume differently by default.

In practice

Snakemake decides what to rerun through five default rerun-triggers: input files and modification times are tracked directly, while code, parameters, and the software environment are tracked by hash. If any of the five changed for a unit, that unit and everything downstream of it reruns; otherwise it is skipped. Resumption needs no flag, since running the same snakemake command again picks up wherever the previous run stopped. The triggers can be narrowed (--rerun-triggers mtime is common when environment hashes cause spurious reruns), but know what the narrowing gives up first.

Nextflow resumes from its work directory, where every task execution is cached under a hash covering the task's inputs, script content, software environment, container, and a session identifier. The practical consequence: running the same nextflow run command again starts from scratch, because a new session gets a new identifier. Resuming is explicit:

nextflow run main.nf -profile cluster -resume

Forgetting -resume after a 20-hour run is a rite of passage; make it the default in your run instructions. The work directory also grows without bound, since every task execution leaves its files there. Clean it periodically, keeping what the current results depend on:

nextflow clean -before <run-name> -f

Both tools share two limits that we should know before trusting them. Neither tracks resources that are not declared as inputs: a unit reading a file it never declared will not rerun when that file changes, and the study silently computes on stale data, which is one more reason to declare boundaries honestly in the contract (Orch1). And incremental execution assumes deterministic outputs; a stochastic unit without a fixed seed produces a different result on every legitimate rerun, so what "resume" preserves is no longer well defined.

Both tools also record basic provenance of what actually ran. Snakemake generates a self-contained HTML report with the executed graph, runtime statistics, and input/output records:

snakemake --report report.html

Nextflow's built-in reports (-with-report, -with-timeline, -with-dag) cover execution; for provenance in standard formats there is the nf-prov plugin, which emits RO-Crate or PROV records, though it needs explicit setup in the configuration. These records are what lets a reviewer, or us in a year, answer which version of which unit produced a given result.

Related

Incremental execution is only as good as the declared contracts (Orch1), and the environment hashes it tracks come from the lock files and images of Iso1 and Iso2. The massflow use case leans on this: an interrupted calibration run skips the expensive completed simulator evaluations on resume.