Appearance
Benchmarking Gaussian Process Emulators for High-Dimensional Problems
This study benchmarks Gaussian process emulator (GPE) implementations against five datasets, focused on high-dimensional inputs and outputs. It was built by a master's student on macOS, working in Python, who needed to apply a GPE method from outside the geohazard domain and benchmark it against existing implementations reproducibly.
A shared preprocessing unit prepares five datasets, spanning synthetic and real-world scenarios generated locally or fetched from Figshare and Zenodo. The study fans these datasets out to twelve emulator implementations running in parallel, built on GPyTorch in Python and RobustGaSP in R, and covering exact inference, deep kernel learning, dimensionality reduction, and multi-output methods. Each emulator unit reads the preprocessed data, runs independently, and writes results; a metrics aggregation unit gathers all outputs and scores them for comparison.
Patterns
The preprocessed datasets fan out across twelve independent emulator implementations, then fan back in to a single aggregation unit: Pattern B. See Workflow Patterns for how this pattern composes with the others.
Recommendations applied
Each emulator unit runs in its own conda environment with a pinned lock file (Iso1), so adding a new emulator introduces no dependency conflict with any existing one. CI regenerates conda lock files automatically whenever an environment specification changes, and a benchmark run triggers on every push to the main branch, with results uploaded as a GitHub artifact (Iso4).
Data at every unit boundary is HDF5, exchanged through the filesystem, so any emulator reads and writes through the same format regardless of language (Int1, Int2). Unlike the two calibration studies above, no unit here exchanges data over a live client-server connection, so this study doesn't need Int3.
The study is automated in both Nextflow and Snakemake (Orch1); both support local and SLURM execution profiles (Orch2), and the Snakemake version skips completed emulator runs on rerun (Orch3). Neither version runs a long-lived service process, so this study doesn't need Orch4 either, unlike the two calibration studies.
The master's student added kPCA-PPGaSP, a GPE method from the literature, as a new unit and compared it against the existing implementations; adding it required no changes to any existing unit. Implementing the study in both Nextflow and Snakemake shows that the approach isn't tied to one workflow manager: both versions follow the same unit contracts and produce consistent results. GPU-accelerated execution is available for the PyTorch-based emulators in the SLURM profile. The study has run on macOS and on a local Linux cluster.
Repository
The study is available in two versions: Nextflow and Snakemake.