The Three Pillars

The approach rests on three pillars. Each addresses a distinct class of problems, and each can be adopted on its own. A study might start with Isolation alone and add the other two as it grows; nothing forces you to take all three at once.

Within each pillar, the approach makes a small number of concrete recommendations. They carry stable identifiers (Iso1 through Orch4) so that use cases, blog posts, and tooling can point at exactly one of them. This page is the map; each identifier links to a page that shows how to apply it.

Isolation

Any computational unit can join a study regardless of its programming language, operating system, and dependencies, without interfering with any other unit. Each collaborator develops their unit in its own self-contained environment.

• Iso1 >>> Package Management: specify each unit's dependencies with conda or mamba and pin them with per-OS, per-architecture lock files.
• Iso2 >>> Containerisation: when a package manager is not enough, package the unit and its entire OS layer into a container image, one image per unit.
• Iso3 >>> HPC: reach Docker-restricted clusters by converting the Docker image to an Apptainer image, keeping a single build definition.
• Iso4 >>> Automation: let CI regenerate lock files and rebuild container images, instead of doing that combinatorial work by hand.

Isolation overview →

Interoperation

Data can flow between computational units regardless of the language each unit is written in or the infrastructure it runs on. Units meet only at their boundaries, and those boundaries use language-agnostic formats and standard protocols.

• Int1 >>> Data Formats: no language-tied formats at unit boundaries; start with CSV or JSON, move to HDF5, netCDF, Zarr, or Parquet as data becomes numerically heavy.
• Int2 >>> Location: exchange through the filesystem for as long as you can; switch to S3 or SFTP only when the study spans infrastructures with no shared filesystem.
• Int3 >>> Bandwidth: for bulk transfers over a network use Zarr; for high-frequency exchange between live units use an HTTP client-server setup.

Interoperation overview →

Orchestration

The study runs reliably end-to-end, even as units are added, collaborators change, and the infrastructure evolves. A scientific workflow management system takes over scheduling, data movement, and recovery from interruption.

• Orch1 >>> Automation: know when a script stops being enough, and express each unit as a contract of inputs, outputs, script, and environment.
• Orch2 >>> Scheduling: let the workflow manager place units on laptops, SLURM clusters, or cloud executors; moving infrastructure becomes a configuration change.
• Orch3 >>> Incremental Execution: resume an interrupted study from the last completed unit instead of restarting from scratch.
• Orch4 >>> Service Processes: run a unit as a long-lived server inside a workflow, which Pattern C requires and most workflow tools do not natively expect.

Orchestration overview →

Reading the map

If you already know where your study hurts, go straight to the pillar overview pages: each opens with four guiding questions that route you to the right recommendation. If you are still working out what your study needs, the workflow patterns page is the better entry point.