Kubernetes Job Scheduling for Bathymetric Processing

Kubernetes job scheduling is the coarse-grained execution model within the deployment and cloud infrastructure workflow: one Job processes one survey tile, holds a large working set in memory, runs for minutes to hours, and exits. Bathymetric gridding is the archetypal memory-heavy batch workload — a single tile of multibeam soundings interpolated to a dense DEM can hold several gigabytes of arrays at peak, and if that peak exceeds the node’s memory the pod is OOMKilled with no traceback. The scheduling contract that keeps this reliable is a small set of Job fields: a resource request sized from the tile’s actual memory profile, a node selector that lands the pod on a memory-optimized instance, and deadlines and retry limits that fail a pathological tile fast instead of burning a node for hours. This page covers that contract, a production Job manifest, a Python submitter that raises on failure, and the failure modes that dominate memory-heavy geospatial batch work.

Reference Job Configuration #

The table captures the Job and Pod fields that govern a bathymetric gridding workload, with the values and rationale for a memory-heavy tile.

The single most consequential number is requests.memory, and it must come from measurement. A float32 DEM tile of $W \times H$ cells occupies $4 W H$ bytes; the interpolation stage typically holds the source points, a KD-tree or distance buffer, and the output grid simultaneously, so peak resident memory runs roughly three to four times the output grid. For a $8000 \times 8000$ tile that is $4 \times 8000^2 \approx 256$ MB of output but a peak near 1 GB during interpolation — and a coarser survey with dense soundings can push an order of magnitude higher. Size the request from a profiled run, not from the output size.

Job Lifecycle and Scheduling Flow #

The diagram traces a tile from submission through scheduling onto a memory-optimized node to a terminal state, showing where the deadline, backoff, and OOM branches divert the flow.

Production Job Manifest #

The manifest below encodes the reference configuration for one indexed gridding Job. It pins the image by digest (the artifact produced in containerizing marine geospatial pipelines with Docker), targets the memory-optimized pool, and bounds itself with a deadline and a modest backoff.

apiVersion: batch/v1
kind: Job
metadata:
  name: bathy-grid-tile-0142
  labels:
    app: bathymetric-gridding
    survey: ne-shelf-2026
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 2
  activeDeadlineSeconds: 5400        # 90 min hard cap
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: bathymetric-gridding
    spec:
      restartPolicy: Never
      nodeSelector:
        node.kubernetes.io/instance-type: r6i.4xlarge
      tolerations:
        - key: workload
          operator: Equal
          value: bathymetry
          effect: NoSchedule
      containers:
        - name: gridder
          image: ghcr.io/agency/marine-pipeline@sha256:<pin-real-digest>
          args: ["--tile", "0142", "--survey", "ne-shelf-2026"]
          env:
            - name: INPUT_URI
              value: s3://agency-marine/surveys/ne-shelf-2026/points.zarr
            - name: OUTPUT_URI
              value: s3://agency-marine/products/ne-shelf-2026/dem_0142.tif
            - name: TARGET_EPSG
              value: "32619"
            - name: AWS_REGION
              value: us-east-1
          resources:
            requests:
              memory: "24Gi"
              cpu: "4"
              ephemeral-storage: "20Gi"
            limits:
              memory: "28Gi"
              ephemeral-storage: "24Gi"
          volumeMounts:
            - name: scratch
              mountPath: /scratch
      volumes:
        - name: scratch
          emptyDir:
            sizeLimit: 20Gi

Python Submitter That Raises on Failure #

Fanning out a survey means creating one Job per tile and watching each to a terminal state. The submitter below uses the kubernetes client to create the Job, then polls Job status and raises a specific exception on any non-success outcome so the orchestrator — whether a Prefect flow from orchestrating marine pipelines with Dask and Prefect or a shell loop — can act on it. It never treats a missing success as success.

import logging
import time
from dataclasses import dataclass

from kubernetes import client, config
from kubernetes.client import ApiException

log = logging.getLogger("marine.k8s")


class JobFailedError(RuntimeError):
    """Raised when a bathymetric Job reaches a terminal non-success state."""


@dataclass(frozen=True)
class JobResult:
    name: str
    succeeded: bool
    reason: str


def _load_config() -> None:
    try:
        config.load_incluster_config()
    except config.ConfigException:
        config.load_kube_config()


def submit_and_wait(
    job: client.V1Job,
    namespace: str = "bathymetry",
    poll_interval_s: float = 10.0,
    watch_timeout_s: float = 6000.0,
) -> JobResult:
    """Create a Job and block until it terminates, raising on failure.

    Success requires an observed `succeeded` count >= the requested completions.
    Any failed pod, deadline breach, or backoff exhaustion raises JobFailedError
    so no caller can mistake an unfinished or failed tile for a completed one.
    """
    _load_config()
    batch = client.BatchV1Api()
    core = client.CoreV1Api()
    name = job.metadata.name

    try:
        batch.create_namespaced_job(namespace=namespace, body=job)
    except ApiException as exc:
        raise JobFailedError(f"Job {name} create failed: {exc.reason}") from exc
    log.info("Submitted Job %s to namespace %s", name, namespace)

    deadline = time.monotonic() + watch_timeout_s
    completions = job.spec.completions or 1

    while time.monotonic() < deadline:
        status = batch.read_namespaced_job_status(name, namespace).status
        if status.succeeded and status.succeeded >= completions:
            log.info("Job %s succeeded", name)
            return JobResult(name=name, succeeded=True, reason="Complete")

        if status.failed:
            reason = _terminal_reason(core, namespace, name)
            log.error("Job %s failed: %s", name, reason)
            raise JobFailedError(f"Job {name} failed: {reason}")

        for cond in status.conditions or []:
            if cond.type == "Failed" and cond.status == "True":
                raise JobFailedError(f"Job {name} condition Failed: {cond.reason}")

        time.sleep(poll_interval_s)

    raise JobFailedError(f"Job {name} did not terminate within {watch_timeout_s:.0f}s")


def _terminal_reason(core: client.CoreV1Api, namespace: str, job_name: str) -> str:
    """Extract a human-actionable reason (OOMKilled, etc.) from the failed pod."""
    pods = core.list_namespaced_pod(
        namespace, label_selector=f"job-name={job_name}"
    ).items
    for pod in pods:
        for cs in pod.status.container_statuses or []:
            term = cs.state.terminated
            if term and term.reason:
                return f"{term.reason} (exit {term.exit_code})"
        if pod.status.reason:
            return pod.status.reason
    return "Unknown terminal reason"

Scheduled Surveys with CronJobs, and Argo / Dask-on-Kubernetes #

Not every bathymetric workload is a one-shot fan-out. Recurring products — a nightly re-grid of an actively surveyed area, a weekly change-detection surface — are naturally expressed as a CronJob, which stamps out a fresh Job on a schedule. The Job template inside a CronJob is identical to the standalone manifest above; the wrapper adds a schedule, a concurrencyPolicy, and a history limit. For memory-heavy gridding the critical field is concurrencyPolicy: Forbid, which prevents a slow run from overlapping its successor and doubling the memory demand on the pool:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: bathy-nightly-regrid
spec:
  schedule: "0 3 * * *"            # 03:00 UTC daily
  concurrencyPolicy: Forbid        # never overlap a slow run with the next
  startingDeadlineSeconds: 600     # skip a missed slot rather than backfill a storm
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 5400
      template:
        spec:
          restartPolicy: Never
          nodeSelector:
            node.kubernetes.io/instance-type: r6i.4xlarge
          containers:
            - name: gridder
              image: ghcr.io/agency/marine-pipeline@sha256:<pin-real-digest>
              resources:
                requests: {memory: "24Gi", cpu: "4"}
                limits: {memory: "28Gi"}

When the survey is too large for a hand-rolled fan-out, two higher-level engines take over while preserving the same per-tile Job contract. Argo Workflows expresses the tile grid as a DAG template, running each tile as a Job step with dependencies, retries, and a fan-in step that assembles the tiles and registers the STAC catalog entry — the workflow controller handles the parallelism withItems over the tile list. Dask-on-Kubernetes (via dask-kubernetes operator) is the opposite shape: instead of one Job per tile, it launches a scheduler pod and a pool of worker pods that share a distributed task graph, so a single logical gridding spreads across workers with disk spill between them. The choice mirrors the coarse-versus-fine orchestration split from the Deployment & Cloud Infrastructure overview: Argo keeps the memory-heavy tile as an isolated Job with a hard limit, while Dask-on-K8s trades that isolation for shared-memory elasticity. The engine-level trade-offs for the bathymetric case specifically are compared in Dask vs Ray for bathymetric processing at scale.

Validation Gates and QC #

Three checks stand between a Job spec and the Kubernetes cluster; each prevents a class of wasted or dangerous run.

1. Resource-limit sanity versus tile size. Before submitting, compute the tile’s expected peak memory from its cell dimensions and reject any Job whose limits.memory is below that peak times a safety factor. This catches the mismatch that produces OOMKilled at run time and turns it into a fast pre-flight failure. The relationship is $\text{peak} \approx k \cdot 4 W H$ bytes with $k \approx 3.5$ for gridded interpolation; a limit under that will OOM.

2. activeDeadlineSeconds present and bounded. A Job without a deadline can pin a memory-optimized node indefinitely if a tile hits a pathological interpolation. The gate rejects any spec missing activeDeadlineSeconds or setting it above a policy ceiling.

3. PodDisruptionBudget for the workload. Long-running gridding pods must not be evicted mid-computation by a routine node drain. A PodDisruptionBudget with maxUnavailable: 0 (or a small value across a fan-out) protects in-flight tiles from voluntary disruption while still permitting involuntary eviction under genuine node failure.

import logging
import math

log = logging.getLogger("marine.k8s.gate")

SAFETY_FACTOR: float = 3.5          # peak / output-grid ratio for interpolation
BYTES_PER_CELL: int = 4             # float32
DEADLINE_CEILING_S: int = 7200      # policy: no tile Job may exceed 2h


def _parse_gi(value: str) -> float:
    if value.endswith("Gi"):
        return float(value[:-2])
    if value.endswith("Mi"):
        return float(value[:-2]) / 1024.0
    raise ValueError(f"Unsupported memory quantity: {value!r}")


def validate_job_spec(width: int, height: int, mem_limit: str, deadline_s: int | None) -> None:
    """Reject a Job spec whose memory limit or deadline is unsafe for the tile.

    Raises before submission so an undersized limit surfaces as a config error,
    not an OOMKilled pod that wasted a node's scheduling slot.
    """
    peak_gib = SAFETY_FACTOR * BYTES_PER_CELL * width * height / (1024 ** 3)
    limit_gib = _parse_gi(mem_limit)
    if limit_gib < peak_gib:
        raise ValueError(
            f"memory limit {limit_gib:.1f}Gi < projected peak {peak_gib:.1f}Gi "
            f"for {width}x{height} tile — will OOM; raise limit or reduce chunk"
        )
    if deadline_s is None:
        raise ValueError("activeDeadlineSeconds missing — unbounded Job rejected")
    if deadline_s > DEADLINE_CEILING_S:
        raise ValueError(f"activeDeadlineSeconds {deadline_s} exceeds ceiling {DEADLINE_CEILING_S}")
    log.info("Job spec OK: peak≈%.1fGi ≤ limit %.1fGi, deadline %ds",
             peak_gib, limit_gib, deadline_s)

Common Failure Modes and Diagnosis #

OOMKilled (exit 137) #

Symptom: The pod terminates with reason OOMKilled and exit code 137, no Python traceback, often minutes into gridding.

Root cause: Peak working set exceeded limits.memory. Common triggers are an undersized request, an eager .compute() that materialized a Dask array, or a denser-than-expected sounding tile.

Diagnosis: kubectl describe pod <pod> shows Reason: OOMKilled; compare the container’s memory limit against the tile’s projected peak from the validation gate. Reduce chunk size, raise the limit onto a larger node, or fix a premature materialization. The full remediation playbook — chunk resizing, spill configuration, and request tuning — is the subject of handling OOM kills in bathymetric Kubernetes Jobs.

ImagePullBackOff #

Symptom: The pod stays Pending/Waiting with reason ImagePullBackOff and never starts.

Root cause: The pinned image digest is unreachable — a private registry without an imagePullSecret, a typo in the digest, or a garbage-collected digest that was never retained.

Diagnosis: kubectl describe pod <pod> shows the pull error and the registry host. Confirm the digest exists (crane manifest <ref>) and that the pod’s service account has the pull secret. Because deployments reference the image strictly by digest per the Deployment & Cloud Infrastructure overview, a missing digest is a retention-policy problem, not a tag drift.

Ephemeral-Storage Eviction #

Symptom: A running pod is evicted with reason Evicted and a message about ephemeral storage; the Job restarts or fails.

Root cause: Dask spill files or a scratch COG written to the node’s local disk exceeded requests.ephemeral-storage, and the kubelet evicted the pod to reclaim disk.

Diagnosis: kubectl describe pod <pod> shows The node was low on resource: ephemeral-storage. Budget ephemeral-storage requests and limits to cover Dask’s spill directory and any scratch outputs, or mount a sized emptyDir at the scratch path (as the manifest does) so spill is bounded and accounted.

Chunk Too Large for Node #

Symptom: Every tile in a survey OOMs regardless of node size, or scheduling stalls with 0/N nodes available: insufficient memory.

Root cause: The gridding chunk is larger than any node in the memory-optimized pool can hold, so no node can ever satisfy the request.

Diagnosis: Compare the projected peak from the validation gate against the pool’s largest instance memory. If peak exceeds the biggest node, the tile must be subdivided — split the survey block into smaller tiles or reduce the interpolation neighborhood. This is fundamentally a partitioning decision; the trade-off between larger nodes and smaller chunks, and how it differs across execution engines, is compared in Dask vs Ray for bathymetric processing at scale.

Pipeline Integration and Downstream Handoff #

A completed Job’s output is a tile of the survey product in object storage; the fan-out across all tiles reconstructs the full DEM. The submitter’s raise-on-failure contract is what lets a higher-level orchestrator treat the survey as a unit: a Prefect flow from orchestrating marine pipelines with Dask and Prefect can submit hundreds of tile Jobs, retry the transient failures, and fail the run only when a tile is deterministically broken. For very large surveys, an Argo Workflow or a Dask-on-Kubernetes cluster replaces the shell loop, but the per-tile Job contract — sized request, deadline, backoff, raise on failure — stays identical.

The science that runs inside each Job is defined in the bathymetric processing and terrain modeling domain: the gridding algorithm, the datum handling, and the IHO uncertainty validation all execute within the container this Job schedules. This scheduling layer’s only job is to give that computation exactly the memory it needs, on the right node, with a hard bound on how long it may take and an unambiguous signal when it fails.

---

Related #

• Handling OOM Kills in Bathymetric Kubernetes Jobs — chunk resizing, spill configuration, and request tuning to stop exit-137 failures
• Containerizing Marine Geospatial Pipelines with Docker — the pinned image this Job pulls by digest
• Orchestrating Marine Pipelines with Dask & Prefect — flow-level orchestration that submits and retries tile Jobs
• Dask vs Ray for Bathymetric Processing at Scale — engine trade-offs behind the chunk-versus-node partitioning decision
• Bathymetric Processing & Terrain Modeling — the gridding science that executes inside each scheduled Job

Up: Deployment & Cloud Infrastructure for Marine Pipelines