Int1 >>> Data Formats

Recommendation. Use language-agnostic interchange formats for all data at unit boundaries, never language-tied ones. Start with text formats (CSV, JSON) while the data is not numerically heavy; once it consists primarily of floating-point values, move to binary formats: HDF5 or netCDF for hierarchical and structured data, Zarr for multi-dimensional arrays, Parquet for tables.

When this applies

At every boundary where one unit's output becomes another unit's input, from the first day of the study. Inside a unit, any format goes; the rule concerns only what crosses between units.

Why

Language-tied formats such as .pkl (Python), .npy (NumPy), or .jld2 (Julia) turn every boundary into a language commitment: a collaborator working in R would need to install and maintain a Python runtime just to read a neighbouring unit's output. Adopting a newly published method in another language then means conversion steps or rewrites, exactly where we want to move fast. With language-agnostic boundaries, adding a unit in a new language never requires touching existing units.

In practice

The rule of thumb is a progression, not a single choice. We start with CSV for tables and JSON for structured metadata; every language reads and writes both, they are inspectable with any text editor, and for configuration files, small result summaries, and metadata they remain the right choice indefinitely.

They stop being right when the data becomes primarily floating-point. Text formats store numbers as strings, and that costs twice. Storage: a 64-bit double needs 17 significant decimal digits to survive a round-trip through text, which means 20 to 24 characters where binary needs 8 bytes, a bloat factor of roughly 2.5 to 3. Correctness: most writers do not emit 17 digits by default, so values silently lose precision on every pass through the boundary. If a unit writes CSV with a default float format and another unit reads it back, the numbers are already not the ones that were computed. For anything numerically heavy, we switch to a binary format, picked by the shape of the data.

For hierarchical and structured data, we use HDF5, a self-describing container of named, typed arrays with attached metadata, with mature libraries in every relevant language (h5py in Python, HDF5.jl in Julia, rhdf5 or hdf5r in R, native APIs in C and Fortran). netCDF builds on HDF5 with conventions for dimensions and coordinates, and is the natural choice where those conventions are established, as in climate and geoscience data, or where your tooling produces it anyway (MCMC output through ArviZ, for instance, is netCDF).

For large multi-dimensional arrays, we use Zarr. It stores an array as chunks with JSON metadata, which makes reading a slice cheap without loading the whole array. That chunking is also what makes Zarr work well over networks, which becomes relevant in Int3.

For tabular data at scale, we use Apache Parquet, a columnar format with typed columns and compression, read natively by pandas, Arrow, R, and Julia. It is the binary successor to CSV: same mental model, none of the string overhead.

Whatever the format, write down the boundary schema in prose next to the code: names, shapes, units of measure, and who writes and who reads each file. The format keeps the bytes portable; the agreed schema is what keeps collaborators aligned, and it is the part of the boundary that no format can enforce for us.

Related

Where these files live and how they move between machines is Int2; when data must flow during execution rather than between executions, serialisation moves into the exchange protocol, which is Int3. A unit's declared inputs and outputs under Orch1 are exactly these boundary files.