Coastal & Marine Spatial Analysis Pipelines

Setting S3 Lifecycle Rules for Survey Archives

A marine data archive grows without bound and is read very unevenly. Lifecycle rules are the mechanism for making that cheap, and they are also silent, delayed and irreversible: a rule that matches the wrong prefix transitions objects to a class you cannot read quickly, and you discover it during the reprocess that needed them. This page sets them deliberately, within the publishing workflow.

Root cause: an archive is three populations, not one

Applying one rule to a bucket treats three different things identically.

Raw soundings and raw AIS. Large, immutable, read rarely — but read entirely when they are read, because a reprocess consumes the whole survey. The right target for cold storage, and the population where retrieval cost matters most.

Derived products. COGs, Zarr stores, density grids. Read constantly by consumers, small relative to the raw data, and regenerable from it. Keeping them in standard storage costs little and saves a reprocess every time someone opens a map.

Staging and temporaries. Rechunker intermediates, preempted tile uploads, failed multipart parts. Should not exist for more than a day and, without a rule, exist forever while being invisible in the console’s object listing.

import boto3


def bucket_composition(bucket: str) -> dict[str, float]:
    """Bytes by top-level prefix, so the populations can be seen."""
    s3 = boto3.client("s3")
    sizes: dict[str, float] = {}
    paginator = s3.get_paginator("list_objects_v2")
    for page in paginator.paginate(Bucket=bucket):
        for obj in page.get("Contents", []):
            top = obj["Key"].split("/", 1)[0]
            sizes[top] = sizes.get(top, 0.0) + obj["Size"]
    return {k: v / 2**40 for k, v in sorted(sizes.items(), key=lambda kv: -kv[1])}

The listing above will not show the fourth population at all. Incomplete multipart uploads are charged for and are not objects, so they appear in no listing and in no size total — only in the bill.

The Fourth Population Appears in No Listing A horizontal composition bar of a survey bucket. Raw soundings and raw AIS occupy the large majority. Derived products occupy a much smaller share. Staging and temporary prefixes occupy a small share that grows steadily because nothing deletes them. A fourth segment, drawn detached and dashed, represents incomplete multipart uploads: it is charged for at the standard storage rate and appears in no object listing and no size total, so it is invisible to every audit that walks the objects. A note records that on a bucket with frequent preemptions this segment can reach several per cent of the bill and that the rule which removes it is two lines and has no downside. what the bucket actually holds raw soundings and raw AIS — 84 TiB derived — 22 TiB staging ? incomplete multipart uploads charged for · in no listing · in no size total Every audit that walks objects misses the dashed segment, because it is not made of objects. On a bucket with frequent preemptions or flaky uploads it reaches several per cent of the bill. The rule that removes it is two lines, applies to the whole bucket, and has no downside worth discussing. It is also the one most often left out, because nothing ever showed it was needed.

Step-by-step

1. The rule that costs nothing 

ABORT_INCOMPLETE = {
    "ID": "abort-incomplete-multipart",
    "Filter": {"Prefix": ""},                       # the whole bucket
    "Status": "Enabled",
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 3},
}

Three days rather than one, so a legitimately slow upload of a large mosaic is not killed mid-flight. There is no scenario in which a five-day-old incomplete upload is wanted.

2. Tier the raw archive on measured access, not on a round number 

def transition_age_from_access(access_log_days: list[int], *, keep_warm_fraction: float = 0.99) -> int:
    """Age beyond which `keep_warm_fraction` of reads no longer occur."""
    if not access_log_days:
        raise ValueError("no access records — do not guess a transition age")
    ordered = sorted(access_log_days)
    idx = int(len(ordered) * keep_warm_fraction)
    age = ordered[min(idx, len(ordered) - 1)]
    logger.info("%.0f%% of reads occur within %d days of an object's creation",
                keep_warm_fraction * 100, age)
    return age

Marine archives have a characteristic access shape: heavy for the weeks around delivery, near zero for a year, then an occasional complete re-read when a model improves or a dispute arises. That means the transition age can be aggressive — ninety days is often defensible — while the retrieval budget has to assume a whole-archive read.

3. Budget the retrieval before choosing the class 

STORAGE_USD_PER_TIB_MONTH = {"STANDARD": 23.6, "STANDARD_IA": 12.9,
                             "GLACIER_IR": 4.1, "DEEP_ARCHIVE": 1.0}
RETRIEVAL_USD_PER_TIB = {"STANDARD": 0.0, "STANDARD_IA": 10.2,
                         "GLACIER_IR": 30.7, "DEEP_ARCHIVE": 20.5}


def class_cost(tib: float, storage_class: str, *, months: int = 36,
               full_reads: int = 2) -> dict[str, float]:
    """Total cost of holding and re-reading an archive over its useful life."""
    if storage_class not in STORAGE_USD_PER_TIB_MONTH:
        raise ValueError(f"unknown class {storage_class}")
    store = tib * STORAGE_USD_PER_TIB_MONTH[storage_class] * months
    retrieve = tib * RETRIEVAL_USD_PER_TIB[storage_class] * full_reads
    return {"storage": store, "retrieval": retrieve, "total": store + retrieve}

For 84 TiB over three years with two full reprocesses, deep archive saves a great deal on storage and costs thousands on retrieval — and, more importantly, imposes a restore latency of hours before the reprocess can start. Instant-retrieval glacier classes usually win for a live archive precisely because the reprocess is a scheduled, whole-archive read rather than a random one.

4. Apply per prefix, and verify the filter 

def put_lifecycle(bucket: str, transition_days: int) -> None:
    """Three rules, one per population. Applied together so they cannot drift."""
    rules = [
        ABORT_INCOMPLETE,
        {"ID": "raw-to-cold", "Status": "Enabled",
         "Filter": {"Prefix": "raw/"},
         "Transitions": [{"Days": transition_days, "StorageClass": "GLACIER_IR"}]},
        {"ID": "staging-expiry", "Status": "Enabled",
         "Filter": {"Prefix": "_staging/"},
         "Expiration": {"Days": 7}},
    ]
    boto3.client("s3").put_bucket_lifecycle_configuration(
        Bucket=bucket, LifecycleConfiguration={"Rules": rules})
    logger.info("applied %d lifecycle rule(s) to %s", len(rules), bucket)


def preview_matches(bucket: str, prefix: str, limit: int = 20) -> list[str]:
    """What a rule's filter would actually match. Run this before applying."""
    s3 = boto3.client("s3")
    page = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=limit)
    keys = [o["Key"] for o in page.get("Contents", [])]
    logger.info("prefix %r matches %d object(s) (showing %d)",
                prefix, page.get("KeyCount", 0), len(keys))
    return keys

Note there is no rule for derived products. Leaving them in standard storage is the deliberate choice: they are a fifth of the volume, read constantly, and moving them saves little while adding retrieval latency to every consumer request.

Storage Is the Visible Half of the Bill Four stacked bars, one per storage class, showing three-year total cost for an 84 tebibyte raw archive with two full reprocess reads. Standard storage has a large storage segment and no retrieval segment. Standard infrequent access roughly halves the storage segment and adds a modest retrieval segment. Glacier instant retrieval cuts storage to about a sixth and adds a larger retrieval segment, giving the lowest total. Deep archive has the smallest storage segment of all but a retrieval segment that brings its total close to glacier instant retrieval, and is annotated with an additional restore latency of several hours before any reprocess can begin. A note states that the latency, not the money, is usually what rules out deep archive for a live archive. 84 TiB raw, 3 years, 2 full reprocesses STANDARD $71k STANDARD_IA $41k GLACIER_IR $17k — lowest DEEP_ARCHIVE $6k + several hours of restore latency pale: storage · dark: retrieval Latency, not money, usually rules out deep archive: a reprocess waits for the restore.

Verification and acceptance test 

import pytest


def test_lifecycle_rules_match_the_intended_prefixes(bucket):
    cfg = boto3.client("s3").get_bucket_lifecycle_configuration(Bucket=bucket)
    ids = {r["ID"]: r for r in cfg["Rules"]}
    assert "abort-incomplete-multipart" in ids, "the free rule is missing"
    assert ids["raw-to-cold"]["Filter"]["Prefix"] == "raw/"
    assert "Transitions" not in ids.get("staging-expiry", {}), (
        "staging should expire, not transition — a cold temporary file is the worst case")


def test_no_derived_product_would_be_transitioned(bucket):
    cfg = boto3.client("s3").get_bucket_lifecycle_configuration(Bucket=bucket)
    for rule in cfg["Rules"]:
        prefix = rule.get("Filter", {}).get("Prefix", "")
        if rule.get("Transitions") and preview_matches(bucket, prefix or "derived/"):
            assert not prefix.startswith("derived"), (
                f"rule {rule['ID']} would move derived products to cold storage")

The second test is the one that catches an empty-prefix rule, which matches everything including the products consumers read every minute — the single most damaging lifecycle mistake, and the easiest to make.

Retention is a policy question wearing a technical costume

Lifecycle rules delete things. That makes the expiry half of this configuration a records-management decision rather than an engineering one, and the two most common failures are both organisational.

Deleting data a regulator expects to exist. Hydrographic surveys supporting charting, dredging contracts and environmental permits often carry statutory retention periods measured in years or decades. An expiry rule written to control cost can quietly breach one, and nothing in the object store knows the difference between a temporary file and a legally-required record.

Keeping everything because nobody will decide. The opposite failure costs money indefinitely and, worse, makes the archive harder to use: a bucket holding every intermediate from every run for eight years is one in which nobody can find the authoritative version of anything.

The workable middle is to make retention explicit per population, in the same file as the lifecycle configuration, with an owner named:

RETENTION = {
    "raw/":       {"years": None, "owner": "hydrographic-lead", "basis": "statutory, indefinite"},
    "derived/":   {"years": 7,    "owner": "data-products",     "basis": "regenerable from raw"},
    "_staging/":  {"years": 0,    "owner": "platform",          "basis": "no value after the run"},
    "manifests/": {"years": None, "owner": "hydrographic-lead", "basis": "provenance for raw"},
}


def rules_from_retention(retention: dict) -> list[dict]:
    """Generate lifecycle rules from the declared policy, so the two cannot drift."""
    rules = [ABORT_INCOMPLETE]
    for prefix, policy in retention.items():
        if policy["years"] == 0:
            rules.append({"ID": f"expire-{prefix.strip('/')}", "Status": "Enabled",
                          "Filter": {"Prefix": prefix}, "Expiration": {"Days": 7}})
        elif policy["years"] is not None:
            rules.append({"ID": f"expire-{prefix.strip('/')}", "Status": "Enabled",
                          "Filter": {"Prefix": prefix},
                          "Expiration": {"Days": int(policy["years"] * 365.25)}})
    return rules

Generating the rules from the policy rather than maintaining both means the question “why does this expire” always has an answer, and the answer names a person. Note that manifests/ never expires even though it is tiny: a run manifest outliving its data is harmless, and data outliving its manifest is an archive nobody can interpret.

Generate the Rules From the Policy, Not Beside It A retention table listing four prefixes with their retention period, owner and stated basis: raw soundings kept indefinitely on a statutory basis owned by the hydrographic lead, derived products kept seven years because they are regenerable and owned by the data-products team, staging expired after a week because it has no value after the run and owned by the platform team, and manifests kept indefinitely because a manifest outliving its data is harmless while data outliving its manifest is uninterpretable. An arrow leads from the table to a generated lifecycle configuration, showing that the rules are derived rather than maintained separately, so the question of why an object expires always has an answer that names a person. prefixkeepownerbasis raw/foreverhydro leadstatutory derived/7 yearsdata productsregenerable _staging/7 daysplatformno value after the run manifests/foreverhydro leadprovenance for raw generated lifecycle rules abort-incomplete · raw-to-cold expire-derived · expire-staging Generated rather than maintained alongside, so “why does this expire?” always has an answer that names a person. manifests/ never expires even though it is tiny: a manifest outliving its data is harmless, and data outliving its manifest is an archive nobody can interpret.

Edge cases and gotchas

  • Transitions have a minimum object size that makes them counter-productive. Objects under 128 KB are charged as 128 KB in some classes, so transitioning a Zarr store of millions of small chunks can increase cost. Tier the raw archive, which is large objects, and leave chunked stores alone.
  • Rules apply to versions too, if versioning is on. A bucket with versioning and no NoncurrentVersionExpiration accumulates every overwritten object forever. On a pipeline that republishes a product nightly, that is a copy per night, indefinitely.
  • Lifecycle actions are asynchronous and irreversible. Transitions happen within a day or so of the rule matching, and moving back costs a retrieval plus a re-upload. Use preview_matches before applying, on the real bucket, every time — the check takes seconds and the mistake takes days to undo.

Up: Publishing COG & Zarr to Object Storage