Int2 >>> Location

Recommendation. Exchange data between units through the filesystem for as long as the study allows it: each unit writes to a path, the next reads from it. Switch to remote storage only when the study spans infrastructures with no shared filesystem, using S3-compatible object storage in general and SFTP when the remote side is a filesystem.

When this applies

Whenever a boundary file from Int1 has to get from the producing unit to the consuming one. The answer depends only on where the units execute, which is why it can change over a study's life without touching the formats.

Why

Filesystem exchange is the simplest mechanism that works: no network dependency, no credentials, no services to keep alive, and trivially debuggable because you can look at the files. Every step beyond it reaches further but adds moving parts. Taking those steps early, "to be safe", is how studies end up maintaining object-store credentials for units that run on the same machine.

In practice

Filesystem exchange covers more ground than it first appears. It works on a single laptop, and it works just as well across the nodes of an HPC cluster, because clusters mount a shared parallel filesystem (Lustre and GPFS are the common ones) that every compute node sees under the same path. A study whose units all run within one cluster never needs anything else.

One caveat on the laptop side. On Windows and macOS, Docker containers run inside a Linux virtual machine, so a containerised unit cannot use the host's inter-process communication (no shared memory, no host Unix sockets). For exchange between a containerised unit and a non-containerised one on the same machine, bind-mounted directories and network protocols are the only reliable channels; the filesystem option here means mounting a host directory into the container:

docker run --rm -v "$PWD/exchange:/exchange" ghcr.io/your-org/forward-model:0.1.0

We move to remote storage only when the study spans separate infrastructures with nothing mounted in common: a laptop and a cloud instance, or two different HPC systems. The general answer there is S3-compatible object storage, reachable from any infrastructure over HTTP with nothing to mount. "S3-compatible" covers far more than AWS: MinIO provides the same API self-hosted, and many universities and research data centres already run an S3 endpoint. Units interact with it through any S3 client; with rclone the pattern stays close to file copying:

rclone copy results/design.csv s3remote:my-study/results/
rclone copy s3remote:my-study/results/ results/

Keep the location out of unit code. A unit that takes input and output paths as arguments does not care whether orchestration hands it a local path or syncs against a bucket before and after; burying bucket names inside units is how location decisions become rewrites.

When the remote side exposes a filesystem rather than an object store, such as a remote workstation or an existing dataset sitting on a server, SFTP is the better fit. It speaks filesystem semantics (paths, directories, permissions) over SSH, which is usually already available:

sftp user@server:/data/observations/2024/ observations/

The distinction stays clean if we keep it simple: S3 when both sides should meet at a neutral, always-on store; SFTP when we reach into a machine that already holds the data.

Related

What is inside the exchanged files is Int1. If units need to exchange data repeatedly while running, no storage location solves that; you need the client-server setup of Int3. Under orchestration, declared input and output paths are how the workflow manager tracks these exchanges (Orch1).