Iso3 >>> HPC

Recommendation. For units that run on HPC clusters, use Apptainer as the container runtime but keep the Dockerfile as the single build definition: build the Docker image first and convert it to an Apptainer .sif image as a final step, rather than maintaining a separate Apptainer definition file.

When this applies

This concerns every containerised unit that needs to run on an HPC cluster. A Docker image alone is sufficient for laptops and cloud environments; the moment a cluster enters the study, this conversion step becomes necessary. If none of your units are containerised, HPC portability is already covered by conda lock files (Iso1) recreated on the cluster.

Why

HPC clusters typically do not allow Docker, because its daemon runs with root privileges, which is unacceptable on shared machines. Apptainer (formerly Singularity) is the container runtime built for this setting: it runs containers as the invoking user, without a daemon, and is what most clusters install instead.

Apptainer can build images from its own definition file format, but those builds have no layer caching: every change, however small, triggers a full rebuild. During development, when the build definition changes often, that is painfully slow. Keeping the Dockerfile as the only build definition gives us Docker's layer caching while we iterate, and one conversion command when we deploy. It also avoids maintaining two build definitions that will inevitably drift apart.

In practice

The pipeline is Dockerfile → Docker image → Apptainer .sif image. The .sif file (Singularity Image Format) is a single self-contained file, which is convenient on clusters: we copy one file and run it.

If the Docker image is published in a registry (Iso2), we convert it directly on the cluster:

apptainer build forward-model.sif docker://ghcr.io/your-org/forward-model:0.1.0

If the image only exists locally, we export it and convert the archive:

docker save ghcr.io/your-org/forward-model:0.1.0 -o forward-model.tar
apptainer build forward-model.sif docker-archive://forward-model.tar

Running the unit on the cluster then looks like:

apptainer exec forward-model.sif julia --project=/app /app/src/forward_model.jl

A few things behave differently from Docker, and it is better to meet them here than on the cluster. Apptainer mounts the home directory and the current working directory into the container by default, whereas Docker isolates the filesystem unless directories are mounted explicitly; use --no-home or explicit --bind mounts if that leaking is a problem. The container runs as your cluster user, not root, so anything in the image that assumes root at run time will fail. And match the image architecture to the cluster: converting an arm64 image for an x86_64 cluster produces an image that will not run. The multi-architecture manifests from Iso2 exist to prevent exactly this.

If you use Snakemake for orchestration, note that Apptainer is its native container runtime: a rule that declares a Docker image gets converted automatically at run time. That convenience does not remove the value of building the .sif ourselves for long-running studies, since we then control when the conversion happens instead of it running at first execution.

Related

The image being converted here is defined and built under Iso2; CI can run the conversion so a current .sif is always available (Iso4). Which executor submits the containerised unit to the cluster's scheduler is an orchestration concern (Orch2).