Orch4 >>> Service Processes

Recommendation. When a unit must run as a long-lived service, as the model server in a Pattern C loop does, use the workflow system's mechanism for it: in Snakemake, declare the unit as a service rule; in Nextflow, which has no native support, use a file-based sentinel and plan for its two failure modes.

When this applies

Exactly when your study contains a Pattern C occurrence: a model server must stay alive while a client unit queries it repeatedly over the Int3 channel. If every unit in your study starts, computes, and exits, this recommendation does not apply.

Why

Workflow management systems assume each unit starts, runs, and exits, and build everything on that assumption: a task is "done" when its process exits successfully, and downstream tasks start after. A server inverts the logic. It is "ready" while still running, must keep running while its consumers execute, and should be shut down, not marked failed, when they finish. Squaring that with a SWMS requires explicit support or a deliberate workaround; improvising here produces workflows that hang without explanation.

In practice

In Snakemake, declare the server's output as a service. A service rule is started before its consumers, is considered ready rather than complete, and is terminated automatically once all consuming rules have finished:

rule model_server:
    output:
        service("server.socket"),
    conda:
        "units/model/environment.yml"
    shell:
        "python serve_model.py --socket {output}"

rule mcmc:
    input:
        "server.socket",
        data="results/observations.csv",
    output:
        "results/posterior.nc",
    conda:
        "units/mcmc/environment.yml"
    shell:
        "python run_mcmc.py --server {input[0]} --data {input.data} --out {output}"

The lifecycle problem is handled for you, which is the argument for Snakemake when Pattern C dominates your study.

Nextflow has no native equivalent, and the established workaround is a file-based sentinel. The server process writes a small file once it is actually listening, and the client process declares that file as an input, so Nextflow's normal dependency machinery delays the client until the server is ready. Write the sentinel only after binding the port, and put the connection details in it, so the client reads where to connect instead of assuming:

# end of server start-up, after the port is bound
with open("server.ready", "w") as f:
    f.write(f"{hostname}:{port}\n")

The client's first action is reading that file and connecting. This works, and both calibration use cases run on it, but it has two failure modes to recognise on sight. If the server crashes before writing the sentinel, the client never starts and the workflow stalls, apparently idle, with no error; the fix is a timeout on the client side or in the workflow configuration, so a missing sentinel becomes a loud failure instead of a silent wait. If the server crashes after the client has started, the client's next request fails with a connection error and the client crashes; that one at least is visible, and a client that retries briefly before giving up separates a server restart from a server death.

Two general rules keep either variant sane. The server must shut down when its consumers are done (Snakemake does this for you; in Nextflow, have the client's completion trigger the shutdown, or run the server with a hard wall-time), because an orphaned server blocks the workflow from ever finishing on a cluster allocation. And treat the sentinel as part of the unit contract: it is a declared output and input like any boundary file, not a hidden convention between two scripts.

Related

The client-server channel this manages is Int3; recognising that your study needs it at all is the patterns page. The sentinel pattern was worked out in the massflow calibration study, where an R surrogate server signals its port to a Python MCMC client. Contracts and their honesty are Orch1.