Setting COG Overviews & Block Size for S3 #

Serving map tiles from a Cloud-Optimized GeoTIFF on S3 should mean a client fetches a few kilobytes with one or two HTTP range requests. When it instead means dozens of small GET calls and multi-second tile latency — and a surprising monthly S3 request bill — the file was almost certainly written with the wrong internal layout. This is the most common defect in the publishing COG & Zarr to object storage workflow: a valid GeoTIFF that is not cloud-optimized, so every zoomed-out view reads full-resolution pixels and every tile straddles many internal blocks. This guide diagnoses the striped-and-overview-less GeoTIFF, then rebuilds it with 512-pixel internal blocks, a proper overview pyramid, and a compression codec matched to bathymetric raster data.

Root cause: striped layout and missing overviews #

A default gdal_translate or rasterio write produces a striped GeoTIFF — pixels stored in full-width scanline strips. To read a 256×256 tile, a client must fetch every strip the tile touches, and each strip spans the entire raster width. Over HTTP range reads on S3 that becomes many partial-object GETs per tile. Two structural properties fix it:

• Internal tiling (block size). Pixels are stored in independent square blocks. A reader fetches only the blocks overlapping the requested tile. If the block size matches the serving tile size (or a clean multiple), a tile becomes one contiguous byte range.
• Overviews (a pyramid). Downsampled copies of the raster at successive zoom levels. Without them, rendering a whole-survey thumbnail reads the full-resolution array; with them, the reader fetches a tiny pre-computed level.

The number of pyramid levels needed to bring the coarsest overview down to roughly one block is $\lceil \log_2(\max(W, H) / B) \rceil$ for a raster of width $W$, height $H$, and block size $B$. A 40 000-pixel-wide survey with 512-pixel blocks needs about seven overview levels.

Reproducing the problem #

rio cogeo validate is the authoritative check. On a plain GeoTIFF it fails loudly:

rio cogeo validate s3://survey-bucket/grids/line_01_plain.tif

The following errors were found:
- The file is greater than 512xH or Wx512, it is recommended to include internal overviews
- The file is not tiled: block size is 40000x1 (striped)
line_01_plain.tif is NOT a valid cloud optimized GeoTIFF

Watching the reader confirms the cost. GDAL logs every range request when curl debugging is on, and a striped file generates a long burst of tiny reads for a single low-zoom tile:

CPL_CURL_VERBOSE=YES CPL_DEBUG=ON \
  gdal_translate -of PNG -srcwin 0 0 256 256 \
  /vsis3/survey-bucket/grids/line_01_plain.tif /tmp/tile.png 2>&1 \
  | grep -c "Range: bytes"
# 60+ range GETs for one tile — each one billed as an S3 request

Step 1 — Choose block size and compression #

Match the internal block size to how tiles are served. For XYZ/WMTS tile servers that emit 256×256 tiles, a 512-pixel internal block is the widely-used default: it keeps the block count (and thus the file’s IFD/offset tables) manageable while still letting a reader fetch a single 256 tile from one block. The compression codec depends on the pixel content:

ZSTD compresses faster and often smaller than DEFLATE at equivalent levels but requires the reader’s GDAL to be built with ZSTD support — safe on modern stacks, worth confirming for older client tooling. Never apply a lossy codec such as WEBP to a raster that will feed DEM interpolation for seafloor mapping or any quantitative analysis; use it only for pre-rendered visual layers. The trade-off between raster and array formats for the analysis stage is covered in understanding NetCDF vs GeoTIFF for marine data.

The diagram shows why block size governs range-read efficiency:

Step 2 — Build the COG with overviews #

rio-cogeo wraps the GDAL COG driver and writes tiled blocks plus overviews in the correct internal order (header and offset tables at the front, so a reader learns the layout from the first range read). Build float bathymetry with a floating-point predictor and bilinear overview resampling:

import logging
from rio_cogeo.cogeo import cog_translate
from rio_cogeo.profiles import cog_profiles

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

def build_bathymetry_cog(src_path: str, dst_path: str) -> None:
    """Write a validated float32 bathymetry COG with 512px blocks and overviews.

    Uses DEFLATE + floating-point predictor (lossless) and bilinear overview
    resampling suited to continuous depth surfaces. Raises on translate failure.
    """
    profile = cog_profiles.get("deflate")
    profile.update(
        blockxsize=512,
        blockysize=512,
        predictor=3,          # 3 = floating-point predictor for float rasters
    )
    config = {
        "GDAL_NUM_THREADS": "ALL_CPUS",
        "GDAL_TIFF_OVR_BLOCKSIZE": "512",   # overview blocks match main blocks
    }
    cog_translate(
        src_path,
        dst_path,
        profile,
        config=config,
        overview_resampling="bilinear",     # continuous data → bilinear, not nearest
        overview_level=None,                # auto: enough levels to reach ~1 block
        web_optimized=False,                # keep native CRS; True reprojects to 3857
        use_cog_driver=True,
        quiet=False,
    )
    logger.info("COG written: %s", dst_path)

build_bathymetry_cog("line_01_plain.tif", "line_01_cog.tif")

Two resampling choices matter. Continuous depth surfaces use bilinear (or average); categorical rasters such as a habitat classification must use nearest so overviews do not invent intermediate class codes. web_optimized=True reprojects to EPSG:3857 for direct XYZ serving — convenient for a basemap, but it destroys the native survey CRS, so keep it False when the COG must stay analytically faithful.

Step 3 — Upload with the correct content type and cache headers #

An object served with the wrong Content-Type breaks some tile clients and CDN caching. Put the COG with image/tiff and a long cache lifetime, since a published survey grid is immutable:

import boto3
from botocore.config import Config

def upload_cog(local_path: str, bucket: str, key: str) -> None:
    """Upload a COG to S3 with correct content type and immutable cache headers."""
    s3 = boto3.client("s3", config=Config(retries={"max_attempts": 5, "mode": "adaptive"}))
    s3.upload_file(
        local_path,
        bucket,
        key,
        ExtraArgs={
            "ContentType": "image/tiff; application=geotiff; profile=cloud-optimized",
            "CacheControl": "public, max-age=31536000, immutable",
        },
    )

upload_cog("line_01_cog.tif", "survey-bucket", "grids/line_01_cog.tif")

Once the raster layer is published this way it sits alongside the array-format products; the companion array store and its metadata pitfalls are covered in fixing Zarr consolidated metadata on GCS.

Step 4 — Validate and measure #

First prove structural correctness, then prove the runtime win. Validation must pass with no errors:

rio cogeo validate line_01_cog.tif

line_01_cog.tif is a valid cloud optimized GeoTIFF

Inspect the block and overview structure programmatically as a regression guard in CI:

import rasterio

def assert_cog_layout(path: str, expected_block: int = 512, min_overviews: int = 3) -> None:
    """Fail fast if a published COG lost its tiling or overviews.

    Raises AssertionError with a diagnostic so a broken publish never ships.
    """
    with rasterio.open(path) as ds:
        bx, by = ds.block_shapes[0]
        assert (bx, by) == (expected_block, expected_block), \
            f"block size {bx}x{by} != expected {expected_block}"
        n_ovr = len(ds.overviews(1))
        assert n_ovr >= min_overviews, \
            f"only {n_ovr} overview levels; need >= {min_overviews}"
    print(f"COG layout OK: {expected_block}px blocks, {n_ovr} overviews")

assert_cog_layout("line_01_cog.tif")

Then re-run the range-read count from the reproduction step against the new object. A correct COG serves a low-zoom tile from a single overview block:

CPL_CURL_VERBOSE=YES gdal_translate -of PNG -srcwin 0 0 256 256 \
  /vsis3/survey-bucket/grids/line_01_cog.tif /tmp/tile_cog.png 2>&1 \
  | grep -c "Range: bytes"
# 2–3 range GETs (header + overview block) versus 60+ before

Dropping from tens of requests to a handful per tile is the acceptance criterion — it directly reduces both latency and S3 GET charges.

Edge cases and gotchas #

• Nodata plus overview resampling contaminates the coast. With average or bilinear resampling, nodata pixels at the survey edge bleed into valid overview pixels unless nodata is masked. Set the source nodata (or an internal mask/alpha band) before building so GDAL excludes it from the average — otherwise coarse zoom levels show a fringe of impossible depths near the shoreline.
• Sparse ocean tiles still cost a request. Even a fully-nodata block over open water is fetched before the reader learns it is empty. For very sparse surveys, a SPARSE_OK=TRUE internal-masked COG lets GDAL skip absent blocks entirely, avoiding wasted GETs on tiles that hold no data.
• Reader-side GDAL settings dominate real latency. A perfectly built COG still performs badly if the client does not set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR (stops a directory listing on open) and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif. Document these for consumers; the file layout is only half of the performance story.

---

Related #

• Fixing Zarr Consolidated Metadata on GCS — the array-store counterpart to COG publishing (sibling)
• Publishing COG & Zarr to Object Storage — parent workflow for cloud-native raster and array delivery (up one level)
• Understanding NetCDF vs GeoTIFF for Marine Data — choosing the right storage format before you publish
• DEM Interpolation Techniques for Seafloor Mapping — the analysis stage that consumes lossless COGs

Up: Publishing COG & Zarr to Object Storage