Int3 >>> Bandwidth

Recommendation. When bandwidth becomes the limiting factor, match the mechanism to the situation. For bulk transfers of large arrays over a network, use Zarr rather than HDF5. For high-frequency exchange between two live units, use an HTTP client-server setup, with JSON for small payloads and a binary serialisation such as MessagePack or Protocol Buffers when numerical payloads grow.

When this applies

In two situations. First, when units exchange large volumes of data in bulk across a network, so transfer time starts dominating. Second, when units exchange data at high frequency during execution, which is the defining trait of Pattern C: an algorithm querying a model thousands of times in a loop. If neither applies, we stay with plain files (Int1, Int2).

Why

Writing a file and reading it back is fine once per unit, but inside an online loop the per-exchange overhead multiplies by the number of iterations. A Bayesian calibration that queries its forward model 100,000 times cannot afford a filesystem round-trip per query, let alone a manual handover; the paper's data-feedforward analysis calls this case physically infeasible without live coupling. Bulk transfer has the mirrored problem: monolithic files force you to move everything to read anything.

In practice

For bulk data over a network, we prefer Zarr over HDF5. Both store arrays; the difference is that Zarr splits an array into separately addressable chunks, so a reader over HTTP or S3 fetches only the chunks it touches, while HDF5 is designed around a local file and generally means downloading the whole thing first. A consumer can slice a Zarr array straight out of object storage:

import zarr
responses = zarr.open("s3://my-study/responses.zarr", mode="r")
subset = responses[:, 0:10]   # fetches only the chunks containing these columns

For high-frequency exchange, we run the two units as a client-server pair over HTTP: the unit holding the expensive computation (typically the model) becomes a server exposing it, and the other unit (typically the algorithm) queries it repeatedly. HTTP carries some overhead per request, but it is point-to-point, cross-language, and supported everywhere, which keeps the language-agnostic boundary property intact: the client neither knows nor cares that the server is R or Julia or Fortran.

Two choices follow. The transport is settled (HTTP); the serialisation format inside the requests is not. JSON is the right start, being readable and universally supported, and for scalar and small-vector payloads its overhead does not matter. It is still a text format though, so the same string-encoding costs as CSV reappear per request, and for large arrays exchanged at frequency they add up. At that point we move the payload to a binary serialisation: MessagePack is a near drop-in binary replacement for JSON, and Protocol Buffers adds typed schemas at the cost of a compilation step.

We rarely need to build this from scratch. UMBridge implements exactly this pattern for model-based uncertainty quantification: the model side wraps its evaluation function in a UMBridge server, the algorithm side calls it like a local function, and HTTP with JSON happens underneath.

import umbridge
model = umbridge.HTTPModel("http://localhost:4242", "forward")
result = model([[0.25, 1.3]])   # a list of parameter vectors in, predictions out

Its limit matters before committing: the UMBridge interface accepts and returns arrays only. If your units need to exchange anything not naturally array-shaped, such as structured metadata or variable-length results, UMBridge does not fit and we write the small HTTP service ourselves.

Either way, the server is now a long-lived process inside the study, and that collides with how workflow managers think about units. That problem, and its solutions, is Orch4.

Related

The escalation path that leads here starts in Int1 and Int2; take this step only when exchange frequency or volume forces it. Running the server side under a workflow manager is Orch4, and identifying whether your study contains this situation at all is what the patterns page is for.