Per-Process Conda Environments in Nextflow

February 2026 · Isolation · Nextflow

---

The Problem

Our initial SHOWME.how workflows shared a single Conda environment across all processes. It worked — until it didn't.

Adding a new unit with a conflicting dependency broke the shared environment. Updating one package for Unit B changed behavior in Unit A. The environment file became a graveyard of pins and workarounds.

This is exactly the problem isolation is supposed to solve. We had violated the one unit, one environment principle.

The Fix: Per-Process Conda Environments

Nextflow's conda directive lets each process declare its own environment:

process preprocess {
    conda 'envs/preprocess.yaml'   // this process only
    // ...
}

process evaluate_r {
    conda 'envs/r-stats.yaml'      // completely separate
    // ...
}

Nextflow creates and caches each environment independently. Processes run in their own isolated Conda env. A dependency conflict in one process has zero impact on any other.

Two Approaches

Option 1: Separate environment.yml per process

envs/
├── preprocess.yaml
├── python-ml.yaml
└── r-stats.yaml

Best for: workflows where units have genuinely different dependencies.

Option 2: Inline Conda spec

process quick_step {
    conda 'python=3.11 numpy scipy'
    // ...
}

Best for: simple, single-purpose steps that don't warrant a full file.

The Tradeoff

Per-process environments take longer to set up initially — Conda resolves each environment separately. But Nextflow caches them: on subsequent runs (or -resume), cached environments are reused instantly.

First run: slow (environment creation) All subsequent runs: fast (cache hit)

Result

Our workflows now follow the SHOWME.how principle strictly: each unit declares its environment alongside its code. Adding or updating a unit never touches another unit's environment. The GP emulator benchmarking use case demonstrates this pattern end-to-end.

→ See it in the GPE benchmarking use case