Deployment & Cloud Infrastructure for Marine Pipelines

Deploying automated coastal and marine geospatial pipelines is fundamentally a reproducibility problem wearing an infrastructure costume. A bathymetric gridding job that ran correctly on an analyst’s laptop must produce byte-identical vertical datum transformations when it runs six months later on an ephemeral cloud node that was provisioned, executed one job, and was destroyed within the hour. That guarantee is hard because marine geospatial code depends on native libraries — GDAL, PROJ, GEOS — whose versions silently change the numeric result of a coordinate transformation, and because the datasets are too large to live on local disk, so every stage reads and writes over the network to object storage. This page defines the operational standards for packaging, scheduling, and publishing agency-scale marine pipelines: the container image as the unit of reproducibility, job orchestration as the execution model, and cloud object storage as the data plane, with the native-library pinning discipline that ties all three together.

The audience here already knows how to compute a DEM or reconstruct a vessel track; the bathymetric processing pipeline and the AIS vessel tracking domain cover that science in depth. What follows is the layer beneath: how to make those pipelines run unattended, at scale, on infrastructure you do not own and cannot inspect after the fact.

Deployment Architecture Overview #

A marine pipeline in production is a directed flow from immutable source data, through a containerized processing image scheduled as one or more orchestrated jobs, into cloud-optimized outputs in object storage, and finally into a catalog that makes those outputs discoverable and auditable. Each boundary in that flow is a contract: the container fixes the software, the orchestrator fixes the compute budget, object storage fixes the data layout, and the catalog fixes the provenance.

The three paradigms below map directly onto the four child topics of this domain. The container image is built and hardened in containerizing marine geospatial pipelines with Docker. Job orchestration splits along two execution models — coarse-grained batch jobs in Kubernetes job scheduling for bathymetric processing, and fine-grained task graphs in orchestrating marine pipelines with Dask and Prefect. The data plane is covered in publishing COG and Zarr to object storage.

Paradigm 1: The Container Image as the Reproducibility Unit #

The container image is the atomic unit of reproducibility for a marine pipeline. It is not merely a deployment convenience — it is the only artifact that pins the entire software surface a coordinate transformation depends on. A pyproj datum shift resolves against PROJ transformation grids that live on disk inside the image; a rasterio warp resolves against the GDAL and GEOS versions linked into the wheel. Change any of those and the numeric output changes, usually below the threshold anyone notices until a regulatory review does. Pinning the image by its sha256 digest — not a mutable tag like latest or even 3.8.0 — is what makes “the same pipeline” a meaningful phrase across a fleet of ephemeral nodes.

Two libraries dominate the reproducibility risk. GDAL governs raster and vector I/O and the warping engine; PROJ governs every coordinate and datum transformation. Their versions must be identical between the machine that developed the pipeline and every node that runs it, because a PROJ minor version bump can ship a revised transformation grid that moves a datum shift by centimetres. The geodetic datums and EPSG management reference explains why those grids matter for marine work, and CRS alignment for coastal GIS projects covers the alignment checks a container must pass before it processes a single tile. The containerization page details the multi-stage build, the base-image matrix, and the smoke test that asserts these versions at container start and refuses to run on a mismatch.

Paradigm 2: Job Orchestration and Scheduling #

A pipeline image does nothing until something schedules it against compute. Marine workloads split cleanly into two orchestration models, and choosing the wrong one wastes either money or memory.

The first model is coarse-grained batch: one job processes one survey block or one time partition, holds a large working set in memory, runs for minutes to hours, and exits. Bathymetric gridding is the canonical case — a single memory-optimized node grids one tile, and the tile either fits in RAM or the node is OOMKilled. This is the domain of Kubernetes Jobs and CronJobs, where the scheduling contract is a resource request, a deadline, and a retry policy. The Kubernetes job scheduling reference covers node selectors for memory-optimized instances, activeDeadlineSeconds, and backoffLimit tuning.

The second model is fine-grained task graphs: thousands of small, interdependent tasks whose dependencies form a DAG the scheduler walks, spilling intermediate results to disk and re-running failed tasks without restarting the whole flow. AIS trajectory reconstruction and Zarr rechunking fit here. This is the domain of Dask and Prefect, covered in orchestrating marine pipelines with Dask and Prefect. The two models compose: a Prefect flow commonly submits Kubernetes Jobs as tasks, so a single deployment uses both.

Paradigm 3: Cloud Object Storage as the Data Plane #

At agency scale the data does not fit on any node’s local disk, so object storage — S3, GCS, Azure Blob — is not a backup target but the working data plane. Every pipeline stage reads its inputs and writes its outputs over the network, and the file formats must be chosen so that a stage can read the bytes it needs without downloading the whole object. Cloud-Optimized GeoTIFF and Zarr exist precisely for this: both expose internal tiling and metadata that let rasterio or xarray issue ranged reads for a spatial window. The publishing COG and Zarr to object storage reference covers the write path — overview generation, block size, and the atomic-completion discipline that prevents partial objects from being read as complete.

The subtle failures in this layer are all about atomicity. Object stores have no rename and no transaction; a Zarr store is a tree of thousands of independent objects, and a job that dies mid-write leaves a store that looks complete but is missing chunks. Every write must therefore be followed by an explicit completion marker that downstream consumers check before they open the data, exactly as the AIS ingestion pipeline uses a per-batch manifest to detect torn writes.

Native Library Versioning Across the Fleet #

The single most common source of “it worked in staging” incidents in marine pipelines is version skew in the native GIS stack. GDAL, PROJ, and GEOS are C/C++ libraries; the Python packages that wrap them (rasterio, pyproj, fiona, shapely) may bundle their own copies as manylinux wheels or link against the system copy from a conda or apt install. When those disagree — a wheel-bundled PROJ 9.3 next to a system PROJ 9.1 with a different grid catalog — a datum transformation can silently resolve differently depending on which library the process happens to load first.

The discipline is threefold. First, pin every native library to an exact version and record the versions in a lock file that the image build consumes. Second, distribute the image by digest so that every node in the fleet runs bytes that are provably identical, not merely tagged the same. Third, assert the versions at container start with a smoke test that raises before any data is touched. The build side of this discipline lives in pinning GDAL and PROJ versions in Docker images; the semantic reason it matters — that EPSG codes and datum ensembles are versioned data, not constants — is the subject of geodetic datums and EPSG management.

A minimal fleet-wide assertion, run as the container entrypoint’s first act, catches skew before it corrupts output:

import logging
import os

import pyproj
import rasterio

log = logging.getLogger(__name__)

# Locked at build time; injected as image labels / env for cross-check.
EXPECTED: dict[str, str] = {
    "proj": "9.4.1",
    "gdal": "3.9.2",
}


def assert_native_versions() -> None:
    """Raise if the running native GIS libraries diverge from the pinned fleet versions.

    Version skew between the build image and the run image silently changes datum
    transformations. This must fail the job, not warn, because a warned-past job
    still writes corrupted coordinates to object storage.
    """
    proj_actual = pyproj.proj_version_str
    gdal_actual = rasterio.gdal_version()

    mismatches: list[str] = []
    if not proj_actual.startswith(EXPECTED["proj"]):
        mismatches.append(f"PROJ expected {EXPECTED['proj']}, running {proj_actual}")
    if not gdal_actual.startswith(EXPECTED["gdal"]):
        mismatches.append(f"GDAL expected {EXPECTED['gdal']}, running {gdal_actual}")

    proj_data = os.environ.get("PROJ_DATA") or os.environ.get("PROJ_LIB")
    if not proj_data or not os.path.isdir(proj_data):
        mismatches.append(f"PROJ_DATA not a directory: {proj_data!r}")

    if mismatches:
        raise RuntimeError(
            "Native GIS library skew detected — refusing to run:\n  "
            + "\n  ".join(mismatches)
        )
    log.info("Native versions OK: PROJ %s, GDAL %s, PROJ_DATA=%s",
             proj_actual, gdal_actual, proj_data)

Production Pipeline Entrypoint #

The following entrypoint is the shape every containerized marine pipeline stage should take: read configuration from the environment, assert native versions, open input from object storage lazily, run one stage, write the output back to object storage, and raise on any failure so the orchestrator can retry or surface the error. It reads a Zarr store from S3 through fsspec/s3fs, applies a stage, and writes a Cloud-Optimized GeoTIFF back with rasterio. Nothing is materialized into RAM that does not have to be, logging is structured, and every failure path raises rather than falling back silently.

import logging
import os
import sys
from dataclasses import dataclass

import numpy as np
import rasterio
import s3fs
import xarray as xr
from rasterio.io import MemoryFile
from rasterio.transform import from_origin

log = logging.getLogger("marine.pipeline")


@dataclass(frozen=True)
class StageConfig:
    """Immutable run configuration resolved from the environment."""
    input_uri: str          # e.g. s3://bucket/surveys/2026/mbes.zarr
    output_uri: str         # e.g. s3://bucket/products/2026/dem_mllw.tif
    variable: str           # Zarr variable to process, e.g. "depth"
    target_epsg: int        # horizontal CRS of the product
    aws_region: str

    @classmethod
    def from_env(cls) -> "StageConfig":
        try:
            return cls(
                input_uri=os.environ["INPUT_URI"],
                output_uri=os.environ["OUTPUT_URI"],
                variable=os.environ.get("VARIABLE", "depth"),
                target_epsg=int(os.environ["TARGET_EPSG"]),
                aws_region=os.environ["AWS_REGION"],
            )
        except KeyError as exc:
            raise RuntimeError(f"Missing required environment variable: {exc}") from exc


def open_zarr_from_s3(cfg: StageConfig) -> xr.Dataset:
    """Open a Zarr store from object storage without downloading it whole."""
    fs = s3fs.S3FileSystem(anon=False, client_kwargs={"region_name": cfg.aws_region})
    store = s3fs.S3Map(root=cfg.input_uri.replace("s3://", "", 1), s3=fs, check=False)
    ds = xr.open_zarr(store, consolidated=True, chunks={})
    if cfg.variable not in ds:
        raise KeyError(
            f"Variable {cfg.variable!r} absent from {cfg.input_uri}; "
            f"available: {list(ds.data_vars)}"
        )
    log.info("Opened %s var=%s shape=%s", cfg.input_uri, cfg.variable,
             dict(ds[cfg.variable].sizes))
    return ds


def run_stage(ds: xr.Dataset, cfg: StageConfig) -> tuple[np.ndarray, rasterio.Affine]:
    """Apply one deterministic processing stage and return an array + geotransform.

    The stage here is a chunk-wise nan-aware hillshade-safe fill; the point is the
    contract, not the algorithm — it must be pure, chunk-aligned, and never load the
    full array eagerly.
    """
    da = ds[cfg.variable]
    da = da.where(np.isfinite(da))            # normalize sentinel nodata to NaN
    filled = da.ffill(dim="x").bfill(dim="x")  # lazy, chunk-preserving

    result = filled.compute().values.astype("float32")  # single materialization
    if result.ndim != 2:
        raise ValueError(f"Expected 2-D grid, got shape {result.shape}")
    if not np.isfinite(result).any():
        raise ValueError("Stage produced an all-NaN grid — refusing to publish.")

    res_x = float(da["x"][1] - da["x"][0])
    res_y = float(da["y"][1] - da["y"][0])
    transform = from_origin(
        float(da["x"].min()) - res_x / 2.0,
        float(da["y"].max()) + abs(res_y) / 2.0,
        res_x, abs(res_y),
    )
    return result, transform


def write_cog_to_s3(
    array: np.ndarray, transform: rasterio.Affine, cfg: StageConfig
) -> None:
    """Write a Cloud-Optimized GeoTIFF to object storage with an atomic marker.

    The object is built in memory as a tiled, overview-bearing COG, uploaded to a
    temporary key, then copied to the final key only after a successful upload so
    that no reader ever observes a half-written product.
    """
    profile = {
        "driver": "GTiff",
        "dtype": "float32",
        "count": 1,
        "height": array.shape[0],
        "width": array.shape[1],
        "crs": rasterio.crs.CRS.from_epsg(cfg.target_epsg),
        "transform": transform,
        "nodata": np.nan,
        "tiled": True,
        "blockxsize": 512,
        "blockysize": 512,
        "compress": "deflate",
        "predictor": 3,
    }
    fs = s3fs.S3FileSystem(anon=False, client_kwargs={"region_name": cfg.aws_region})
    final_key = cfg.output_uri.replace("s3://", "", 1)
    tmp_key = final_key + ".partial"

    with MemoryFile() as mem:
        with mem.open(**profile) as dst:
            dst.write(array, 1)
            dst.build_overviews([2, 4, 8, 16], rasterio.enums.Resampling.average)
            dst.update_tags(ns="rio_overview", resampling="average")
        with fs.open(f"s3://{tmp_key}", "wb") as out:
            out.write(mem.read())

    fs.copy(f"s3://{tmp_key}", f"s3://{final_key}")
    fs.rm(f"s3://{tmp_key}")
    log.info("Published COG to %s (%d x %d)", cfg.output_uri, array.shape[1], array.shape[0])


def main() -> int:
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s [%(levelname)s] %(name)s:%(lineno)d %(message)s",
    )
    cfg = StageConfig.from_env()
    log.info("Stage start input=%s output=%s epsg=%d",
             cfg.input_uri, cfg.output_uri, cfg.target_epsg)

    ds = open_zarr_from_s3(cfg)
    array, transform = run_stage(ds, cfg)
    write_cog_to_s3(array, transform, cfg)

    log.info("Stage complete")
    return 0


if __name__ == "__main__":
    try:
        sys.exit(main())
    except Exception:
        log.exception("Pipeline stage failed — non-zero exit for orchestrator retry")
        sys.exit(1)

The structure matters more than the specific stage. Configuration is resolved once and is immutable. Object-store access is lazy at open and materializes exactly once. Every degenerate result — a missing variable, an all-NaN grid, a wrong-rank array — raises rather than writing a plausible-looking but wrong product. And the top-level handler exits non-zero on any exception, which is the signal a Kubernetes Job or Prefect task uses to decide whether to retry.

Failure Modes and Silent Corruption Patterns #

These five failure vectors account for the majority of production incidents when marine pipelines move from a workstation to cloud infrastructure.

1. Version skew between build and run images #

Symptom: A pipeline produces outputs that differ by centimetres to metres from the validated reference, with no code change and no error.

Root cause: The image running in production was rebuilt from a mutable base tag, or the orchestrator pulled a re-pushed tag, so PROJ or GDAL is a different version than the one the pipeline was validated against. The revised PROJ grid catalog shifts the datum transformation.

Diagnosis: Pull the exact image digest the failing job ran (kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].imageID}') and diff pyproj.proj_version_str and rasterio.gdal_version() against the reference image. Any difference is the cause. Enforce the assert_native_versions() entrypoint check so this fails loudly at start instead of silently in the output.

2. OOM kills on ephemeral nodes #

Symptom: A gridding or rechunking job dies with exit code 137 and no Python traceback; the pod shows OOMKilled.

Root cause: The working set — a full-resolution tile plus interpolation scratch — exceeds the node’s memory, or an eager .compute() materialized a Dask array that was supposed to stay lazy. Ephemeral nodes are often smaller than the workstation the code was written on.

Diagnosis: Correlate the tile dimensions against the container memory limit; a float32 grid needs 4 · width · height bytes plus roughly 3–4× that in transient scratch during interpolation. Reduce chunk size or request a memory-optimized node. The Kubernetes job scheduling page covers the resource-request sizing and node selectors that prevent this.

3. Silent partial writes to object storage #

Symptom: A downstream consumer reads a Zarr store or COG and gets a plausible array with a rectangular block of NaN or a truncated extent; no error is raised anywhere.

Root cause: Object stores have no atomic multi-object commit. A job that died after writing some Zarr chunks but before writing the rest, or before writing consolidated metadata, leaves a store that opens successfully but is incomplete.

Diagnosis: Compare the chunk count on disk against the expected count from the array shape and chunking, or check for the absence of a completion marker object. Never let a consumer open a store without first confirming its _SUCCESS (or equivalent) marker exists — the write path in publishing COG and Zarr to object storage writes that marker only after all chunks are flushed.

4. Missing PROJ transformation grids in the container #

Symptom: A datum transformation that worked locally returns coordinates offset by a metre or more in the container, or pyproj logs “grid not found” and falls back to a lower-accuracy path.

Root cause: The PROJ grid files (the proj-data package or a downloaded subset) were not copied into the image, or PROJ_DATA points at an empty directory. pyproj silently degrades to the best available transformation instead of raising.

Diagnosis: Inside the container, run pyproj.datadir.get_data_dir() and confirm the grid .tif files are present. Set PROJ_NETWORK=OFF and force a known grid-dependent transformation; if the accuracy drops, the grids are missing. The container smoke test in containerizing marine geospatial pipelines with Docker asserts grid availability at start.

5. Credential and region misconfiguration #

Symptom: The job hangs and then times out on the first object-store read, or fails with an opaque AccessDenied despite correct-looking IAM policy.

Root cause: The pod’s service account lacks the object-store role, the bucket is in a different region than the client is configured for (triggering slow cross-region redirects), or a PROJ_NETWORK fetch is being blocked by egress rules and stalling the transformation.

Diagnosis: Test the exact credential path from inside the pod with a minimal s3fs/boto3 head_object call and log the resolved region. A cross-region mismatch shows up as high first-byte latency; an IAM gap shows up immediately as AccessDenied. Pin the client region to the bucket region explicitly rather than relying on environment defaults.

Archival, Provenance, and Downstream Handoff #

A published product that cannot be traced back to its inputs and its software environment is not agency-grade, regardless of how correct it is. Every pipeline run must emit two artifacts alongside the data: a STAC catalog entry that makes the product discoverable and a provenance manifest that makes it auditable.

The STAC Item records the spatial and temporal extent, the asset URIs (the COG or Zarr store), and the CRS, so that downstream consumers — the bathymetric terrain modeling workflows that build habitat and dredge-volume products, and the AIS route automation analytics that consume gridded environmental layers — can query for exactly the coverage they need without scanning the bucket. The provenance manifest records the resolved image digest, the pinned GDAL and PROJ versions, the input object checksums, and the run parameters, so a regulator or a future analyst can reconstruct precisely how the product was made.

Handoff artifact reference #

import hashlib
import json
import logging
from datetime import datetime, timezone

import pyproj
import rasterio
import s3fs

log = logging.getLogger("marine.provenance")


def write_provenance_manifest(
    output_uri: str,
    image_digest: str,
    input_uris: list[str],
    target_epsg: int,
    aws_region: str,
) -> None:
    """Emit a signed provenance manifest as a sidecar to the published product.

    The manifest is the audit contract: it captures the exact software environment
    and input identities so any run can be reproduced or challenged. It is written
    only after the product object exists, guaranteeing manifest and data agree.
    """
    fs = s3fs.S3FileSystem(anon=False, client_kwargs={"region_name": aws_region})

    input_checksums: dict[str, str] = {}
    for uri in input_uris:
        with fs.open(uri, "rb") as fh:
            input_checksums[uri] = hashlib.sha256(fh.read()).hexdigest()

    manifest = {
        "produced_at": datetime.now(timezone.utc).isoformat(),
        "product_uri": output_uri,
        "image_digest": image_digest,
        "proj_version": pyproj.proj_version_str,
        "gdal_version": rasterio.gdal_version(),
        "target_epsg": target_epsg,
        "inputs": input_checksums,
    }
    manifest_uri = output_uri.rsplit(".", 1)[0] + ".provenance.json"
    with fs.open(manifest_uri, "w") as out:
        out.write(json.dumps(manifest, indent=2))
    log.info("Provenance manifest written to %s", manifest_uri)

The manifest and the STAC Item are the closing contract of the deployment: the container fixed the software, the orchestrator fixed the compute, object storage fixed the data layout, and these two files fix the record. A pipeline that emits all four contracts on every run is one that an agency can operate unattended and defend under audit.

---

Related #

• Containerizing Marine Geospatial Pipelines with Docker — multi-stage builds, base-image selection, and a start-up smoke test that pins GDAL, PROJ, and PROJ grid availability
• Kubernetes Job Scheduling for Bathymetric Processing — memory-optimized Jobs and CronJobs, resource requests, deadlines, and OOM-resistant scheduling
• Publishing COG & Zarr to Object Storage — overview generation, block size, consolidated metadata, and atomic completion markers
• Orchestrating Marine Pipelines with Dask & Prefect — task-graph execution, retries, and composing Dask work inside Prefect flows
• Geodetic Datums & EPSG Management — why EPSG codes and datum ensembles are versioned data that must be pinned across the fleet

Up: Home