Publishing COG & Zarr to Object Storage

Publishing Cloud-Optimized GeoTIFF (COG) and Zarr products to object storage is the export stage that turns finished marine rasters and gridded arrays into artifacts a browser, tile server, or Dask cluster can read without downloading the whole file, and it sits directly under Deployment & Cloud Infrastructure for Marine Pipelines. A bathymetric DEM, an AIS density raster, or a time-varying sea-surface-temperature grid becomes useful to downstream consumers only when its bytes are laid out so an HTTP range request can fetch exactly the tile or chunk it needs. Get the layout wrong and every failure is silent and expensive: a COG with no overviews forces a tile server to read the full-resolution image for a zoomed-out map; a Zarr store without consolidated metadata turns a single dataset open into thousands of separate GET requests against the bucket listing; a mistyped content-type makes a CDN refuse to cache. This page covers the write path end to end — how to build both formats correctly, how to stream them straight to S3 or GCS, how to validate that the object is genuinely cloud-optimized, and how to diagnose the layout failures that only surface under production read load.

Reference Configuration #

The table below captures the layout parameters and object metadata that govern a production publish. COG parameters control internal tiling and the overview pyramid; Zarr parameters control chunk shape and metadata consolidation; the storage-class and HTTP header rows apply to both.

The single most important rule that spans both formats: write once, never mutate in place, and only advertise the object after it is fully landed and validated. Object stores are eventually consistent for listings, so a consumer that discovers a store mid-write will read a torn artifact. The publish sequence is always: write data objects, validate, write the manifest last.

Write-Path Architecture #

The diagram traces a finished array from memory through format encoding to the object store, and shows how a consumer later fetches only the bytes it needs with a range read.

Memory-Constrained Python Implementation #

The publisher below writes both a COG and a Zarr store directly to a bucket. The COG path builds a tiled, overviewed GeoTIFF and streams it through an fsspec file object so nothing hits local disk; the Zarr path rechunks lazily and writes with consolidated metadata. Both raise on any partial-write condition rather than leaving a torn object behind. For the deep specifics of overview decimation and block-size selection see Setting COG Overviews & Block Size for S3, and for the metadata step that most often goes wrong on Google Cloud see Fixing Zarr Consolidated Metadata on GCS.

import hashlib
import logging
from typing import Sequence

import fsspec
import numpy as np
import rasterio
import xarray as xr
from rasterio.io import MemoryFile
from rasterio.enums import Resampling
from rasterio.shutil import copy as rio_copy

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",
)
logger = logging.getLogger(__name__)

COG_BLOCKSIZE: int = 512
COG_OVERVIEWS: Sequence[int] = (2, 4, 8, 16, 32)
ZARR_SPATIAL_CHUNK: int = 512


class PublishError(RuntimeError):
    """Raised when a published object fails to land completely or validate."""


def _md5(data: bytes) -> str:
    return hashlib.md5(data).hexdigest()


def publish_cog(
    array: np.ndarray,
    transform: rasterio.Affine,
    crs: str,
    dst_uri: str,
    *,
    nodata: float,
    storage_options: dict | None = None,
) -> str:
    """
    Encode ``array`` (shape (bands, y, x)) as a Cloud-Optimized GeoTIFF and
    stream it to ``dst_uri`` (e.g. s3://bucket/key.tif). Overviews are built
    in-memory before the copy so the written object is single-pass readable.
    Returns the MD5 of the written bytes. Raises PublishError on a short write.
    """
    if array.ndim != 3:
        raise PublishError(f"expected (bands, y, x), got shape {array.shape}")

    resampling = Resampling.average
    profile = {
        "driver": "GTiff",
        "dtype": array.dtype.name,
        "count": array.shape[0],
        "height": array.shape[1],
        "width": array.shape[2],
        "crs": crs,
        "transform": transform,
        "nodata": nodata,
        "tiled": True,
        "blockxsize": COG_BLOCKSIZE,
        "blockysize": COG_BLOCKSIZE,
        "compress": "DEFLATE",
        "predictor": 3 if np.issubdtype(array.dtype, np.floating) else 2,
    }

    # Build overviews inside a MemoryFile, then copy with the COG driver so the
    # overview pyramid and IFD layout are ordered for range reads.
    with MemoryFile() as src_mem:
        with src_mem.open(**profile) as src:
            src.write(array)
            src.build_overviews(list(COG_OVERVIEWS), resampling)
            src.update_tags(ns="rio_overview", resampling="average")
        with MemoryFile() as cog_mem:
            rio_copy(
                src_mem.name,
                cog_mem.name,
                driver="COG",
                blocksize=COG_BLOCKSIZE,
                compress="DEFLATE",
                overview_resampling="average",
            )
            payload = cog_mem.read()

    checksum = _md5(payload)
    of = fsspec.open(dst_uri, "wb", **(storage_options or {}))
    with of as fh:
        written = fh.write(payload)
    if written != len(payload):
        raise PublishError(
            f"short COG write to {dst_uri}: {written} of {len(payload)} bytes"
        )
    logger.info("published COG %s (%d bytes, md5=%s)", dst_uri, len(payload), checksum)
    return checksum


def publish_zarr(
    ds: xr.Dataset,
    dst_uri: str,
    *,
    storage_options: dict | None = None,
) -> str:
    """
    Write ``ds`` to a Zarr store at ``dst_uri`` (e.g. gcs://bucket/prod.zarr)
    with spatial chunking aligned to tile reads and consolidated metadata.
    Returns the store URI. Raises PublishError if consolidation is absent.
    """
    if not {"y", "x"}.issubset(ds.dims):
        raise PublishError(f"dataset missing y/x dims: {tuple(ds.dims)}")

    chunks = {"y": ZARR_SPATIAL_CHUNK, "x": ZARR_SPATIAL_CHUNK}
    if "time" in ds.dims:
        chunks["time"] = 1
    ds = ds.chunk(chunks)

    # Encode explicit chunk + compressor so readers get uniform block sizes.
    import numcodecs

    compressor = numcodecs.Blosc(cname="zstd", clevel=5, shuffle=numcodecs.Blosc.SHUFFLE)
    encoding = {
        var: {"compressor": compressor, "chunks": tuple(ds[var].chunk().values())}
        for var in ds.data_vars
    }

    mapper = fsspec.get_mapper(dst_uri, **(storage_options or {}))
    ds.to_zarr(
        mapper,
        mode="w",
        encoding=encoding,
        consolidated=True,
        compute=True,
    )

    if ".zmetadata" not in mapper:
        raise PublishError(f"consolidated metadata missing after write to {dst_uri}")
    logger.info("published Zarr %s (%d vars, consolidated)", dst_uri, len(ds.data_vars))
    return dst_uri

Two design points carry the memory guarantees. The COG builder holds one full copy of the encoded image in a MemoryFile; for products larger than available RAM, write the COG to a temporary local path with build_overviews and upload with a multipart streamed put instead of buffering the whole payload. The Zarr writer never materializes the array — ds.chunk(...) keeps the graph lazy and to_zarr(compute=True) streams each Dask block to the store as it is computed, so peak memory is a small multiple of one chunk, not the full grid.

Validation Gates and Quality Control #

An object is not published until it passes four gates. Advertising an unvalidated store is the root cause of nearly every downstream range-read failure.

1. Cloud-optimized structure. For a COG, run the GDAL validate_cloud_optimized_geotiff check: it confirms the IFDs are ordered, the image is tiled, and overviews are present and internal. For Zarr, confirm the .zmetadata key exists and parses.

2. Object existence and byte count. After the write, HEAD the object (or list the store keys) and assert the returned Content-Length equals the byte count you wrote. A mismatch means a truncated multipart upload.

3. Checksum. Compare the MD5 (or the store’s ETag for single-part puts) against the checksum computed at write time. This catches corruption in transit that a byte count alone misses.

4. Consolidated metadata completeness. For Zarr, open the store with consolidated=True and assert every expected variable and dimension is present; a partial consolidation silently drops arrays written after the .zmetadata snapshot.

from rio_cogeo.cogeo import cog_validate


def validate_cog(dst_uri: str, expected_md5: str, storage_options: dict | None = None) -> None:
    """Assert the published COG is valid, complete, and byte-identical."""
    with fsspec.open(dst_uri, "rb", **(storage_options or {})) as fh:
        payload = fh.read()
    if _md5(payload) != expected_md5:
        raise PublishError(f"COG checksum mismatch at {dst_uri}")

    with MemoryFile(payload) as mem:
        is_valid, errors, warnings = cog_validate(mem.name)
    if not is_valid:
        raise PublishError(f"COG not cloud-optimized: {errors}")
    for w in warnings:
        logger.warning("COG advisory %s: %s", dst_uri, w)
    logger.info("COG validated %s", dst_uri)


def validate_zarr(dst_uri: str, expected_vars: set[str], storage_options: dict | None = None) -> None:
    """Assert consolidated metadata is present and lists every expected variable."""
    mapper = fsspec.get_mapper(dst_uri, **(storage_options or {}))
    if ".zmetadata" not in mapper:
        raise PublishError(f"missing .zmetadata at {dst_uri}")
    ds = xr.open_zarr(mapper, consolidated=True)
    missing = expected_vars - set(ds.data_vars)
    if missing:
        raise PublishError(f"Zarr {dst_uri} missing variables: {missing}")
    logger.info("Zarr validated %s (%d vars)", dst_uri, len(ds.data_vars))

The read-amplification metric is what makes overviews and chunking worth the effort. For a viewport that covers a fraction of the grid, the ratio of bytes actually transferred to bytes logically requested should stay near 1. Define it as

$$A = \frac{B_{\text{transferred}}}{B_{\text{requested}}}, \qquad \text{alert when } A > 4 \text{ over a rolling window}$$

An amplification above 4 means readers are pulling far more than the tile they asked for — the signature of a missing overview level or a chunk shape mismatched to the access pattern. Instrument the tile server and Dask workers to emit transferred versus logical bytes so the ratio can be scraped and alerted on.

Common Failure Modes and Diagnosis #

Missing Overviews Force Full-Resolution Range Reads #

Symptom: A zoomed-out web map is slow and the tile server transfers hundreds of megabytes for a single low-zoom tile; read amplification spikes.

Root cause: The GeoTIFF was written tiled but without an internal overview pyramid, so a low-zoom request has no decimated level to read and must fetch and downsample the full-resolution tiles under the viewport.

Diagnosis: Run gdalinfo on the object and check for Overviews: lines under each band; if absent, or if they appear as external .ovr sidecars rather than internal, the COG is not cloud-optimized. cog_validate reports this as an error. Rebuild with build_overviews before the COG copy, as in publish_cog. Overview decimation and resampling choice are covered in Setting COG Overviews & Block Size for S3.

Un-consolidated Zarr Metadata Triggers Thousands of GETs #

Symptom: xr.open_zarr(...) takes tens of seconds and the bucket access log shows one GET per array .zarray/.zattrs key — thousands of requests before a single chunk is read.

Root cause: The store was written with consolidated=False (or a later append invalidated the snapshot), so the reader must walk the key hierarchy and issue a separate request for every metadata object.

Diagnosis: List the store root and check for a .zmetadata key. If it is missing, re-consolidate with zarr.consolidate_metadata(mapper) and always open with consolidated=True. On GCS this interacts with the store’s flat namespace in a specific way documented in Fixing Zarr Consolidated Metadata on GCS.

Wrong Content-Type Breaks CDN Caching and Range Handling #

Symptom: A COG behind a CDN is never cached; every tile request reaches the origin bucket, and some browsers refuse to honor range requests.

Root cause: The object was uploaded with a default content-type of application/xml or binary/octet-stream instead of image/tiff, so the CDN treats it as uncacheable and clients mishandle partial content.

Diagnosis: HEAD the object and inspect Content-Type and Cache-Control. Re-upload with ContentType="image/tiff" and CacheControl="public, max-age=31536000, immutable" in the put arguments (pass via storage_options/fsspec metadata). Content-addressed products never mutate, so an immutable long TTL is safe.

Region or Endpoint Misconfiguration and Eventual-Consistency Partial Listings #

Symptom: Writes succeed but reads intermittently 404, or a store listing returns only some chunks immediately after a write; cross-region reads are slow and occasionally time out.

Root cause: The client is configured for the wrong region or a legacy endpoint, so requests are redirected or hit a stale listing view; separately, listing a store immediately after writing can return an incomplete key set on eventually-consistent buckets.

Diagnosis: Verify the bucket’s region against the client’s region_name/endpoint_url and prefer the regional endpoint. Never drive a consumer off a live list immediately after write — publish a manifest as the last object and have consumers key off the manifest, so they only ever open a store that is fully landed. Treat a missing manifest as “not ready,” not “empty.”

Pipeline Integration and Downstream Handoff #

The publisher runs as the terminal stage of a marine processing job, most often orchestrated alongside the gridding and cleaning stages described in Orchestrating Marine Pipelines with Dask & Prefect. The upstream stages emit an in-memory or lazily-chunked array; this stage encodes, writes, and validates it, then appends an entry to the run manifest. Because it produces the artifacts other systems read, its correctness guarantees propagate: a validated COG with overviews keeps tile-server latency flat, and a consolidated Zarr store keeps Dask dataset opens cheap.

Two format decisions upstream shape what this stage produces. Whether a given product is better served as Zarr or NetCDF is weighed in Zarr vs NetCDF for Cloud-Native Coastal Pipelines; the tradeoff hinges on cloud range-read access versus single-file portability. Whether a raster product should ship as a COG or a NetCDF grid is the subject of Understanding NetCDF vs GeoTIFF for Marine Data, which explains why continuous multi-band imagery favors COG while multi-dimensional time series favor the array formats.

Each published product is accompanied by a manifest written last, after all data objects have landed and validated:

{
  "product_id": "bathy-dem/gulf-of-maine/2026-07-13",
  "cog": {
    "uri": "s3://cms-products/bathy/gom_2026-07-13.tif",
    "blocksize": 512,
    "overviews": [2, 4, 8, 16, 32],
    "md5": "9f2c1a7e0b4d3c5e6a8f1029384756ab",
    "bytes": 418324992,
    "content_type": "image/tiff"
  },
  "zarr": {
    "uri": "gcs://cms-products/sst/gom_2026-07-13.zarr",
    "chunks": {"time": 1, "y": 512, "x": 512},
    "consolidated": true,
    "variables": ["sst", "sst_anomaly"]
  },
  "crs": "EPSG:32619",
  "written_at": "2026-07-13T06:22:04Z"
}

A consumer reads the manifest first, confirms consolidated is true and the COG md5 matches, and only then opens the product. The manifest is the contract that lets tile servers and Dask clusters trust that every byte they range-read is part of a complete, immutable artifact. For deployment specifics — running the publisher as a scheduled batch job and sizing its memory — see Kubernetes Job Scheduling for Bathymetric Processing, and for the container image that carries a matched GDAL/PROJ stack see Containerizing Marine Geospatial Pipelines with Docker. Authoritative layout references are the COG specification and the Zarr storage specification.

---

Related #

• Setting COG Overviews & Block Size for S3 — decimation levels, resampling, and block-size selection for range-read performance
• Fixing Zarr Consolidated Metadata on GCS — diagnosing and repairing missing .zmetadata on Google Cloud Storage
• Orchestrating Marine Pipelines with Dask & Prefect — the flow that drives this publish stage after gridding
• Zarr vs NetCDF for Cloud-Native Coastal Pipelines — choosing the array format this stage writes
• Understanding NetCDF vs GeoTIFF for Marine Data — when a raster product should ship as a COG versus a gridded NetCDF

Up: Deployment & Cloud Infrastructure for Marine Pipelines