Fixing Zarr Consolidated Metadata on GCS #

Opening a Zarr store on Google Cloud Storage that takes twenty seconds — or raises KeyError: '.zmetadata' outright — is a metadata problem, not a data problem, and it is the second most common failure in the publishing COG & Zarr to object storage workflow after malformed COGs. A Zarr group scatters one tiny .zarray and .zattrs object per array and per group across the bucket. On a local disk, reading a few hundred of these is free; over the network to GCS, each is a separate authenticated HTTP request, and xarray must list the bucket prefix to discover them before it reads any. The fix is metadata consolidation: collapse every scattered metadata key into one .zmetadata object that a single GET retrieves. This guide reproduces the slow open, applies consolidation correctly, and — critically — keeps .zmetadata valid after the appends and region writes that silently make it stale.

Root cause: unconsolidated or stale metadata #

When a store is written without consolidation, opening it forces the client to enumerate keys. Each array contributes at least a .zarray (shape, chunks, dtype, compressor) and a .zattrs; a coastal reanalysis cube with dozens of variables and coordinates can hold hundreds. The open cost scales with that count:

$$T_{\text{open}} \approx T_{\text{list}} + N_{\text{meta}} \cdot t_{\text{GET}}$$

where $N_{\text{meta}}$ is the number of metadata objects and $t_{\text{GET}}$ the per-request round trip to GCS. Consolidation replaces the $N_{\text{meta}}$ term with a single read, so $T_{\text{open}}$ collapses to roughly $2\,t_{\text{GET}}$ regardless of how many arrays the store holds.

There are two distinct symptoms with the same cure but different triggers:

• Never consolidated. The store was written with plain to_zarr(...) and no consolidation call, so .zmetadata does not exist. Opening with consolidated=True raises KeyError; opening with the default emits a slow-open warning and lists the whole prefix.
• Consolidated then mutated. The store was consolidated, but a later append or region write added or resized arrays without refreshing .zmetadata. The stale file now describes the old shape — reads return truncated data or the client misses newly appended chunks entirely. This is the more dangerous case because it fails silently rather than loudly.

Reproducing the slow open #

Point xarray at an unconsolidated store and the delay (and warning) are immediate:

import time
import logging
import xarray as xr

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

t0 = time.perf_counter()
ds = xr.open_zarr("gs://survey-bucket/reanalysis/coastal_cube.zarr")  # no consolidated=True
logger.info("open took %.1f s", time.perf_counter() - t0)
# UserWarning: Failed to open Zarr store with consolidated metadata,
#   falling back to try reading non-consolidated metadata. ...
# open took 21.7 s

Forcing consolidated mode on a store that was never consolidated fails loudly instead:

xr.open_zarr("gs://survey-bucket/reanalysis/coastal_cube.zarr", consolidated=True)
# KeyError: '.zmetadata'

The diagram contrasts the two open paths:

Step 1 — Consolidate metadata after writing #

After all arrays are written, call zarr.consolidate_metadata on the store mapper. xarray’s to_zarr can do this in one shot with consolidated=True, which is the preferred path for a fresh write:

import xarray as xr
import gcsfs

def write_consolidated(ds: xr.Dataset, gcs_path: str, project: str) -> None:
    """Write a dataset to a GCS Zarr store with consolidated metadata.

    consolidated=True writes .zmetadata as the final step so a single GET
    serves the whole store on open. Raises on auth or write failure.
    """
    fs = gcsfs.GCSFileSystem(project=project, token="google_default")
    store = fs.get_mapper(gcs_path)
    ds.to_zarr(store, mode="w", consolidated=True)

write_consolidated(ds, "survey-bucket/reanalysis/coastal_cube.zarr", project="coastal-ops")

For a store written array-by-array by a lower-level pipeline, consolidate explicitly once writing completes:

import zarr
import gcsfs

fs = gcsfs.GCSFileSystem(project="coastal-ops", token="google_default")
store = fs.get_mapper("survey-bucket/reanalysis/coastal_cube.zarr")
zarr.consolidate_metadata(store)   # writes/overwrites .zmetadata from current keys

Step 2 — Open consolidated #

Always open with consolidated=True in production so a missing .zmetadata is a loud error rather than a silent slow fallback. Making the failure explicit means a broken publish is caught in CI, not in a dashboard timeout:

import xarray as xr

ds = xr.open_zarr(
    "gs://survey-bucket/reanalysis/coastal_cube.zarr",
    consolidated=True,
    storage_options={"project": "coastal-ops", "token": "google_default"},
)

This raster-adjacent array store publishes alongside the tiled GeoTIFFs; the block-size and overview concerns for those are covered in setting COG overviews & block size for S3, and the decision of which format to publish in the first place is weighed in Zarr vs NetCDF for cloud-native coastal pipelines.

Step 3 — Re-consolidate after appends and region writes #

This is the step most pipelines miss. Appending a new time slice or writing into an existing region mutates the store’s .zarray shapes and chunk complement, but it does not update .zmetadata. The consolidated file goes stale, and readers using it see the old shape. Re-consolidate as the final action of every mutation:

import xarray as xr
import zarr
import gcsfs
import logging

logger = logging.getLogger(__name__)

def append_timestep(new_ds: xr.Dataset, gcs_path: str, project: str) -> None:
    """Append along the time dimension, then re-consolidate metadata.

    Skipping the re-consolidation leaves .zmetadata describing the pre-append
    shape, so consolidated readers silently miss the new data. Raises on failure.
    """
    fs = gcsfs.GCSFileSystem(project=project, token="google_default")
    store = fs.get_mapper(gcs_path)
    # append WITHOUT consolidated=True (append + consolidate must be sequential)
    new_ds.to_zarr(store, mode="a", append_dim="time", consolidated=False)
    # refresh .zmetadata so consolidated opens see the new time slice
    zarr.consolidate_metadata(store)
    logger.info("appended %d steps and re-consolidated %s", new_ds.sizes["time"], gcs_path)

The same rule applies to region= writes that fill pre-allocated slots: even though the shapes do not change, run consolidate_metadata after the batch so the store is guaranteed consistent for any reader that opens mid-way.

Step 4 — Configure gcsfs for consistency and auth #

Two gcsfs settings prevent subtle failures. First, an explicit project and token avoid falling back to anonymous access — which silently fails on private buckets or, worse, hangs on requester-pays buckets. Second, cache_timeout controls how long directory listings are cached: the default caches listings for the filesystem’s lifetime, so a long-lived reader process can miss objects written after it first listed the prefix. Set a short timeout (or 0 to disable) for readers that must see fresh appends:

import gcsfs

fs = gcsfs.GCSFileSystem(
    project="coastal-ops",
    token="google_default",     # or a service-account JSON path
    cache_timeout=60,           # seconds; low value so appends become visible
    requester_pays=False,       # set True (and project=payer) for requester-pays buckets
)

Step 5 — Verify #

Prove both that the open is fast and that .zmetadata matches the actual arrays — a stale consolidated file opens fast but lies. Time the open and assert consistency:

import time
import zarr
import gcsfs
import xarray as xr

def verify_consolidated(gcs_path: str, project: str, max_open_s: float = 3.0) -> None:
    """Assert .zmetadata exists, is fast to open, and matches the array keys.

    Raises AssertionError if the store is unconsolidated or the consolidated
    file is stale relative to the arrays actually present in the store.
    """
    fs = gcsfs.GCSFileSystem(project=project, token="google_default")
    store = fs.get_mapper(gcs_path)
    assert ".zmetadata" in store, ".zmetadata missing — store was never consolidated"

    t0 = time.perf_counter()
    ds = xr.open_zarr(store, consolidated=True)
    elapsed = time.perf_counter() - t0
    assert elapsed < max_open_s, f"open took {elapsed:.1f}s (> {max_open_s}s) — check consolidation"

    # cross-check: every data/coord var must appear in consolidated metadata
    group = zarr.open_consolidated(store)
    consolidated_keys = set(group.array_keys())
    expected = set(ds.data_vars) | set(ds.coords)
    missing = expected - consolidated_keys
    assert not missing, f"stale .zmetadata: arrays present but not consolidated: {missing}"
    print(f"consolidated OK: opened in {elapsed:.2f}s, {len(consolidated_keys)} arrays match")

verify_consolidated("survey-bucket/reanalysis/coastal_cube.zarr", project="coastal-ops")

A sub-second open and a matching key set together are the acceptance criterion; either alone can hide the stale-metadata trap.

Edge cases and gotchas #

• Appending along a dimension always invalidates consolidation. Any mode="a" write with an append_dim changes an array’s shape, so the previous .zmetadata is immediately wrong. Treat “append then consolidate” as one atomic operation in your pipeline; never let another process open the store consolidated between the two calls, or wrap them so the store is only advertised as ready after re-consolidation.
• Region writes need consolidation even when shapes are unchanged. Filling pre-allocated region= slots does not resize arrays, so the consolidated file stays technically valid — but concurrent writers can leave .zattrs (e.g. a _ARRAY_DIMENSIONS fix or an updated fill value) out of sync. Re-consolidate after the write batch to guarantee every reader sees identical attributes.
• Zarr v3 changes where metadata lives. Zarr v3 replaces the separate .zarray/.zattrs/.zmetadata scheme with a single zarr.json per node and a different consolidation mechanism. A store written with zarr-python v3 in the v3 format will not expose a .zmetadata key, so a v2-style consolidated open fails; pin the Zarr format version explicitly (zarr_format=2 for v2 tooling) until every reader in your fleet supports v3, and confirm the writer and readers agree.

---

Related #

• Setting COG Overviews & Block Size for S3 — the raster counterpart to array-store publishing (sibling)
• Publishing COG & Zarr to Object Storage — parent workflow for cloud-native data delivery (up one level)
• Zarr vs NetCDF for Cloud-Native Coastal Pipelines — choosing the array format before you publish it

Up: Publishing COG & Zarr to Object Storage