Coastal & Marine Spatial Analysis Pipelines

Handling OOM Kills in Bathymetric Kubernetes Jobs

A bathymetric gridding container that dies mid-run with exit code 137 is one of the most common — and most misdiagnosed — failures in the Kubernetes Job scheduling for bathymetric processing workflow. The Job restarts, burns through its backoffLimit, and marks itself Failed, but nothing in the application log points at a bug: the last line is usually a healthy progress message immediately before the kernel terminates the process. Exit 137 is 128 + 9, meaning the container received SIGKILL, and on a memory-limited pod the source of that signal is almost always the Linux out-of-memory killer reacting to the cgroup memory limit being breached. This guide reproduces the failure, explains why gridding workloads hit it, and walks through the memory-budgeting and streaming fixes that keep peak resident memory under the pod ceiling.

Why gridding jobs get OOMKilled

The root cause is rarely a leak. It is that bathymetric gridding is genuinely memory-hungry and the pipeline materializes more of the array than the pod limit allows. Five patterns account for nearly every case:

  • Whole-array materialization. Calling xr.open_dataset(...).load() or numpy.loadtxt on a full survey pulls every soundings row into RAM at once. A 40 000 × 40 000 float64 grid is 12.8 GB before any interpolation buffer exists.
  • A Dask chunk larger than the pod limit. Dask sizes its peak working set from a small multiple of the chunk size. If a single chunk is 3 GB and the worker runs several tasks concurrently, transient memory can exceed a 4 GB limit even though the “average” looks fine.
  • Dask worker nanny accounting. The nanny watches process RSS, not cgroup usage. Under a container limit it can be slow to react, so the kernel OOM killer fires first and takes down the whole pod rather than just pausing a task.
  • NumPy peak versus steady memory. Interpolation and resampling routines allocate temporary copies. A scipy.interpolate.griddata call briefly holds the input points, the query mesh, and the output surface simultaneously — peak is often two to three times steady state.
  • No requests/limits at all. With no memory requests, the scheduler over-packs the node; with no limits, one pod’s growth starves its neighbours and the node-level OOM killer picks a victim that may not even be the offender.

Reproducing the failure

Start by confirming the kill is an OOM event rather than a crash or an eviction. The signature lives in the pod status, not the log:

kubectl describe pod bathy-grid-abc123 | sed -n '/Last State/,/Restart Count/p'
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 13 Jul 2026 09:14:02 +0000
      Finished:     Tue, 13 Jul 2026 09:16:41 +0000
    Restart Count:  3

Reason: OOMKilled with Exit Code: 137 is definitive for a cgroup-limit kill. To see the kernel’s own record — useful when the container was killed by the node-level killer rather than its own cgroup — read dmesg on the node:

kubectl debug node/gke-pool-mem-xyz -it --image=busybox -- dmesg -T | grep -i "killed process"
[Tue Jul 13 09:16:41 2026] Memory cgroup out of memory: Killed process 21874 (python3)
  total-vm:9438208kB, anon-rss:4188160kB, file-rss:20992kB

anon-rss near the container limit confirms the Python process itself grew past the ceiling. If instead you see file-rss dominating or an eviction event in kubectl get events, jump to the edge cases below — the cause is different.

Step 1 — Right-size chunks to fit the pod limit

The first fix is arithmetic: pick chunk dimensions so the peak working set fits inside the limit with headroom for temporaries and the interpreter. Budget for peak, not average — a safe rule is that a Dask worker needs roughly chunk_bytes × concurrent_tasks × 3 for interpolation-heavy stages. Solve for the chunk size that fits your limit:

import logging
import math

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

def chunk_edge_for_limit(
    limit_bytes: int,
    dtype_bytes: int = 8,
    concurrent_tasks: int = 2,
    peak_multiplier: float = 3.0,
    headroom: float = 0.6,
) -> int:
    """Largest square chunk edge (pixels) whose peak working set fits the pod limit.

    headroom reserves capacity for the interpreter, page cache and Dask overhead.
    Raises ValueError if the limit is too small to hold even a 256-pixel chunk.
    """
    usable = limit_bytes * headroom
    bytes_per_chunk = usable / (concurrent_tasks * peak_multiplier)
    edge = int(math.sqrt(bytes_per_chunk / dtype_bytes))
    if edge < 256:
        raise ValueError(
            f"limit {limit_bytes/1e9:.1f} GB too small for a viable chunk; "
            "raise the memory limit or reduce concurrency"
        )
    logger.info("chosen chunk edge: %d px (%.0f MB/chunk)", edge, edge * edge * dtype_bytes / 1e6)
    return edge

# 4 GiB pod limit → safe square chunk edge
edge = chunk_edge_for_limit(limit_bytes=4 * 1024**3)

Feed that edge to open_dataset/rioxarray so Dask never builds an oversized block:

import rioxarray

dem = rioxarray.open_rasterio(
    "s3://survey-bucket/grids/line_01.tif",
    chunks={"x": edge, "y": edge},
    lock=False,
)

Step 2 — Set requests equal to limits

A pod with requests far below limits gets Burstable QoS: the scheduler only reserves the request, so the node can be overcommitted and your Job competes for memory that may not physically exist. For a batch gridding Job you want Guaranteed QoS — set requests equal to limits so Kubernetes reserves real capacity and your pod is the last thing evicted under node pressure. Add generous ephemeral-storage room for Dask spill (see Step 3):

apiVersion: batch/v1
kind: Job
metadata:
  name: bathy-grid
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 5400
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        cloud.google.com/gke-nodepool: mem-optimized
      containers:
        - name: grid
          image: registry.example.org/bathy-grid:2026.07
          resources:
            requests:
              memory: "8Gi"
              cpu: "2"
              ephemeral-storage: "20Gi"
            limits:
              memory: "8Gi"
              cpu: "2"
              ephemeral-storage: "20Gi"

backoffLimit: 2 stops the Job hammering the Kubernetes cluster with doomed retries when the cause is deterministic, and activeDeadlineSeconds caps a hung run so it cannot occupy a memory-optimized node indefinitely.

Step 3 — Tune Dask worker memory targets and spill

Dask can page data to disk before the kernel intervenes, but only if its memory thresholds are set below the cgroup limit and it knows the real limit. Two things matter: tell the worker its limit explicitly (auto-detection under cgroups v1 is unreliable), and set the target/spill/pause fractions so it spills at 60% and pauses at 80% — well clear of the hard ceiling.

from dask.distributed import Client, LocalCluster

cluster = LocalCluster(
    n_workers=2,
    threads_per_worker=1,          # 1 thread caps concurrent tasks → predictable peak
    memory_limit="3.5GiB",         # per-worker, below the 8Gi pod limit across 2 workers
    processes=True,
)
client = Client(cluster)

Set the spill thresholds through Dask config (env vars work well inside a container image):

export DASK_DISTRIBUTED__WORKER__MEMORY__TARGET=0.6   # start spilling to disk
export DASK_DISTRIBUTED__WORKER__MEMORY__SPILL=0.7    # spill aggressively
export DASK_DISTRIBUTED__WORKER__MEMORY__PAUSE=0.8    # stop accepting new tasks
export DASK_DISTRIBUTED__WORKER__MEMORY__TERMINATE=0.95
export DASK_TEMPORARY_DIRECTORY=/scratch              # backed by ephemeral-storage

Spilling trades speed for survival: a job that would OOM now runs slower but finishes. When your surveys routinely need more than one node’s worth of memory, the right move is to distribute the work — compare the two dominant frameworks in Dask vs Ray for bathymetric processing at scale before committing.

Step 4 — Stream tiles instead of loading the whole grid

The most durable fix removes the requirement for a large resident array altogether. Process the DEM as windowed tiles so peak memory is bounded by one tile plus its interpolation halo, independent of the survey’s total size. The diagram contrasts the two memory profiles:

Memory Profile: Whole-Array Load Versus Windowed Tile Streaming Left panel shows resident memory for a whole-array load climbing in a single ramp that crosses the dashed pod memory limit line, where an OOM-kill marker terminates the run. Right panel shows the streaming approach as a low sawtooth: each tile allocates and frees, so the peak never approaches the same dashed limit line and the run completes. Whole-array load memory time pod limit OOMKilled (137) Windowed tile streaming time pod limit peak per tile ≈ one window + halo bounded memory, run completes 
import logging
import numpy as np
import rasterio
from rasterio.windows import Window
from scipy.ndimage import gaussian_filter

logger = logging.getLogger(__name__)

def grid_by_tiles(src_path: str, dst_path: str, tile: int = 2048, halo: int = 64) -> None:
    """Filter/regrid a large DEM window by window so peak memory stays bounded.

    The halo overlaps neighbouring tiles so edge pixels are computed with full
    context, then trimmed on write to avoid seams. Raises on CRS/nodata mismatch.
    """
    with rasterio.open(src_path) as src:
        if src.nodata is None:
            raise ValueError("source DEM has no nodata value; refusing to guess")
        profile = src.profile.copy()
        profile.update(tiled=True, blockxsize=512, blockysize=512, compress="deflate")
        with rasterio.open(dst_path, "w", **profile) as dst:
            for row in range(0, src.height, tile):
                for col in range(0, src.width, tile):
                    r0, c0 = max(row - halo, 0), max(col - halo, 0)
                    r1 = min(row + tile + halo, src.height)
                    c1 = min(col + tile + halo, src.width)
                    win = Window(c0, r0, c1 - c0, r1 - r0)
                    arr = src.read(1, window=win).astype("float32")
                    mask = arr == src.nodata
                    filled = np.where(mask, np.nan, arr)
                    smoothed = gaussian_filter(np.nan_to_num(filled, nan=0.0), sigma=1.0)
                    smoothed[mask] = src.nodata
                    # trim halo back to the core tile before writing
                    tr, tc = row - r0, col - c0
                    core = smoothed[tr:tr + min(tile, src.height - row),
                                    tc:tc + min(tile, src.width - col)]
                    dst.write(core, 1, window=Window(col, row,
                                                     core.shape[1], core.shape[0]))
            logger.info("tiled gridding complete: %s", dst_path)

For a fully orchestrated version where each tile becomes a retryable task with its own memory budget, see orchestrating marine pipelines with Dask & Prefect, and for the interpolation stage the halo protects, DEM interpolation techniques for seafloor mapping.

Step 5 — Verify under load

A fix is only proven when the Job runs to completion with observed headroom. Re-run and watch live memory against the limit:

kubectl apply -f bathy-grid-job.yaml
kubectl wait --for=condition=complete job/bathy-grid --timeout=90m &
watch -n 5 'kubectl top pod -l job-name=bathy-grid --no-headers'
bathy-grid-9f2c1   1780m   5921Mi     # peak stays under the 8Gi limit

Then confirm a clean exit and no restart churn:

kubectl get pod -l job-name=bathy-grid -o jsonpath='{.items[0].status.containerStatuses[0].state.terminated.exitCode}'
# expect: 0
kubectl get job bathy-grid -o jsonpath='{.status.succeeded}'
# expect: 1

Peak resident memory sitting comfortably below the limit — not merely finishing once — is the acceptance criterion, because a run that finishes at 99% of the limit will OOM the moment a survey is slightly denser.

Edge cases and gotchas

  • Off-heap allocations are invisible to Dask’s nanny. GDAL’s block cache and any JVM-based tool (some tide or geoid services) allocate outside the Python heap, so the nanny reports healthy RSS while the cgroup still climbs. Cap GDAL explicitly with GDAL_CACHEMAX=256 (MB) and size Dask limits assuming that cache is added on top, not counted within.
  • Page-cache pressure can trigger the killer under memory.limit_in_bytes. Heavy sequential reads of COGs on cgroups v1 count file-backed pages toward the limit, so a job with modest anon memory still gets OOMKilled. Move to cgroups v2 nodes (which separate reclaimable page cache) or reduce read-ahead, and confirm with the dmesg file-rss value from the reproduction step.
  • Ephemeral-storage eviction masquerades as an OOM. When Dask spills to /scratch and that volume shares the node’s ephemeral storage, exceeding the ephemeral-storage limit produces an Evicted pod with a similar mid-run death — but kubectl get events shows Evicted: ephemeral-storage rather than OOMKilled. Raise ephemeral-storage limits or mount a dedicated volume for spill; do not chase it as a memory problem.

Up: Kubernetes Job Scheduling for Bathymetric Processing