Coastal & Marine Spatial Analysis Pipelines

Handling AIS Timestamp Drift & Clock Skew

Problem Framing

An AIS position report does not carry a full timestamp — it carries a single number from 0 to 59. Everything else about when a message happened is inferred, and that inference is where tracks fall apart. A vessel track built naively from a multi-receiver feed shows negative time deltas, speeds of 4,000 knots between consecutive points, and positions that jump backward in time; this is the timestamp-drift problem at the heart of the Real-Time AIS Stream Ingestion Pipelines workflow. AIS message types 1, 2, and 3 encode only the UTC second-of-minute in which the position was valid. The minute, hour, and date come from whichever receiver logged the message, and receivers disagree: one has a good NTP sync, another drifted 8 seconds, a satellite adds 60–90 seconds of backhaul latency. This page reconstructs a defensible full-UTC timestamp from the second-of-minute plus receiver time, estimates and removes per-receiver skew, and enforces monotonic per-vessel ordering with a reorder buffer so downstream speed & heading profiling never sees a negative dt.

The diagram shows the reconstruction: the payload’s second-of-minute is authoritative for the seconds field, the receiver’s clock supplies the coarse minute/hour/date, and a rollover rule reconciles the boundary case where the receiver’s minute has ticked over but the payload second has not.

Reconstructing Full UTC From AIS Second-of-Minute Left column: payload second-of-minute box (0 to 59, authoritative for seconds). Middle-left: receiver clock box supplying minute, hour, date. These combine in a merge box with a rollover rule handling the 59 to 00 minute boundary. The merge output passes through a skew-correction box that subtracts the estimated per-receiver offset, producing a corrected UTC timestamp box on the right. payload second 0–59 (authoritative) receiver clock minute · hour · date merge + rollover handle :59 → :00 − receiver skew estimated offset corrected UTC ts Then a per-MMSI reorder buffer enforces monotonic ordering downstream.

Root Cause: A Second Is Not a Timestamp

The core defect: AIS types 1/2/3 carry a 6-bit second field (0–59; values 60–63 signal “not available” or manual/dead-reckoning modes). To place the message in absolute time you must attach it to a minute, and the only minute available in a real-time stream is the receiver’s arrival clock. That produces three distinct failure signatures.

Non-monotonic per-MMSI series. When two receivers with skewed clocks interleave, a vessel’s points arrive out of order:

# One MMSI, two receivers, skewed clocks — note the backward jump
# arrival_time (receiver clock)      payload second
# 2026-07-13T09:14:07Z  SHORE-A      07
# 2026-07-13T09:14:15Z  SHORE-B      15   (SHORE-B clock is +8s fast)
# 2026-07-13T09:14:08Z  SHORE-A      08   <-- earlier than the SHORE-B row above

Minute-rollover corruption. A message with payload second 59 logged by a receiver whose clock has just ticked to :00 of the next minute gets stamped a full minute in the future. Future timestamps. A receiver with wrong NTP can stamp messages seconds or minutes ahead of real UTC, poisoning any “latest position” query.

Step-by-Step Fix

Step 1 — Reconstruct full UTC with rollover handling

Combine the receiver’s arrival time (truncated to the minute) with the payload’s authoritative second. The rollover rule: if the payload second is near 59 but the receiver minute rounded down would place it ahead of arrival by more than half a minute, borrow one minute back; symmetrically for the near-0 case:

from __future__ import annotations

import logging
import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s — %(message)s")
logger = logging.getLogger("ais.clock")


def reconstruct_utc(arrival: pd.Series, payload_second: pd.Series) -> pd.Series:
    """Reconstruct full UTC from receiver arrival time and AIS second-of-minute.

    `arrival` must be tz-aware datetime64 in UTC; `payload_second` an integer 0-59.
    Raises ValueError on tz-naive input (timezone contamination is the #1 silent bug).
    """
    if arrival.dt.tz is None:
        raise ValueError("arrival timestamps are tz-naive — refuse to guess UTC")
    sec = payload_second.to_numpy()
    if np.any((sec < 0) | (sec > 59)):
        bad = payload_second[(payload_second < 0) | (payload_second > 59)]
        raise ValueError(f"payload_second out of 0-59 range (types 60-63 not stripped): {bad.tolist()[:5]}")

    minute_floor = arrival.dt.floor("min")
    candidate = minute_floor + pd.to_timedelta(sec, unit="s")
    delta = (candidate - arrival).dt.total_seconds().to_numpy()

    # candidate more than 30 s AHEAD of arrival -> payload belongs to previous minute
    candidate = candidate.mask(delta > 30, candidate - pd.Timedelta(minutes=1))
    # candidate more than 30 s BEHIND arrival -> payload belongs to next minute
    candidate = candidate.mask(delta < -30, candidate + pd.Timedelta(minutes=1))
    return candidate

Step 2 — Estimate and remove per-receiver skew

Each receiver has a roughly constant offset from true UTC over a short window. Estimate it as the median difference between the receiver’s arrival time and the reconstructed UTC across many messages (median is robust to the outliers that drift creates), then subtract it:

def estimate_receiver_skew(df: pd.DataFrame, min_samples: int = 30) -> dict[str, float]:
    """Median arrival-minus-reconstructed offset per receiver, in seconds.

    Receivers with fewer than `min_samples` messages are skipped (skew unreliable).
    """
    skew: dict[str, float] = {}
    diff = (df["arrival"] - df["utc"]).dt.total_seconds()
    for rx_id, idx in df.groupby("rx_id").groups.items():
        vals = diff.loc[idx]
        if len(vals) < min_samples:
            logger.warning("receiver %s: %d samples < %d — skipping skew estimate",
                           rx_id, len(vals), min_samples)
            continue
        skew[str(rx_id)] = float(vals.median())
    logger.info("estimated skew (s): %s", {k: round(v, 2) for k, v in skew.items()})
    return skew


def apply_skew_correction(df: pd.DataFrame, skew: dict[str, float]) -> pd.DataFrame:
    """Subtract each receiver's estimated skew from the reconstructed UTC."""
    corr = df["rx_id"].map(skew).fillna(0.0)
    df = df.copy()
    df["utc"] = df["utc"] - pd.to_timedelta(corr, unit="s")
    return df

Step 3 — Enforce monotonic per-MMSI ordering with a reorder buffer

Even after skew correction, network jitter delivers points slightly out of order. A bounded reorder buffer holds recent points per MMSI, sorts within the buffer window, and emits in order — dropping any point that arrives later than an already-emitted timestamp (a violation the buffer cannot repair without unbounded memory):

from collections import defaultdict, deque


class PerMmsiReorderBuffer:
    """Emit AIS points per MMSI in monotonic UTC order within a bounded lookback.

    Points older than the last emitted timestamp for their MMSI are dropped and
    counted; a large drop count signals skew correction failed upstream.
    """

    def __init__(self, lookback_s: float = 5.0) -> None:
        self._lookback_s = lookback_s
        self._buffers: dict[int, deque] = defaultdict(deque)
        self._last_emitted: dict[int, pd.Timestamp] = {}
        self.dropped = 0

    def push(self, mmsi: int, utc: pd.Timestamp, payload: dict) -> list[dict]:
        buf = self._buffers[mmsi]
        buf.append((utc, payload))
        emitted: list[dict] = []
        horizon = utc - pd.Timedelta(seconds=self._lookback_s)
        # everything older than the horizon is now safe to release, in order
        ready = sorted((t, p) for t, p in buf if t <= horizon)
        for t, p in ready:
            buf.remove((t, p))
            last = self._last_emitted.get(mmsi)
            if last is not None and t < last:
                self.dropped += 1
                continue
            self._last_emitted[mmsi] = t
            emitted.append(p)
        return emitted

Verification

Assert every per-MMSI series is monotonically increasing and no inter-arrival delta implies an impossible speed:

def assert_monotonic_and_plausible(df: pd.DataFrame, max_knots: float = 60.0) -> None:
    """Raise if any MMSI has non-monotonic UTC or an implied speed above max_knots."""
    for mmsi, grp in df.sort_values("utc").groupby("mmsi"):
        ts = grp["utc"]
        if not ts.is_monotonic_increasing:
            raise AssertionError(f"MMSI {mmsi}: non-monotonic timestamps after correction")
        dt = ts.diff().dt.total_seconds().to_numpy()[1:]
        if np.any(dt <= 0):
            raise AssertionError(f"MMSI {mmsi}: non-positive time delta remains")
        # crude great-circle speed sanity check
        lat = np.radians(grp["lat"].to_numpy())
        lon = np.radians(grp["lon"].to_numpy())
        dlat = np.diff(lat); dlon = np.diff(lon)
        a = np.sin(dlat / 2) ** 2 + np.cos(lat[:-1]) * np.cos(lat[1:]) * np.sin(dlon / 2) ** 2
        dist_nm = 3440.065 * 2 * np.arcsin(np.sqrt(a))   # nautical miles
        knots = dist_nm / (dt / 3600.0)
        if np.any(knots > max_knots):
            raise AssertionError(f"MMSI {mmsi}: implied speed {knots.max():.0f} kn > {max_knots}")
    logger.info("monotonic + plausible-speed checks passed for %d vessels", df["mmsi"].nunique())

The great-circle distance uses the haversine formula, $d = 2r,\arcsin!\sqrt{\sin^2(\Delta\varphi/2) + \cos\varphi_1\cos\varphi_2\sin^2(\Delta\lambda/2)}$, with r=3440.065r = 3440.065 nautical miles.

Edge Cases and Gotchas

  • Minute rollover at :59 → :00. The ±30 s masks in Step 1 assume message latency well under 30 seconds — true for terrestrial feeds but not for satellite backhaul. For an S-AIS feed, widen the reconstruction to use the provider’s own capture timestamp when present rather than your ingestion arrival clock, and never mix the two receiver-time sources in one skew estimate.
  • Receivers with wrong NTP and leap seconds. A receiver drifting linearly will show a slowly changing skew that a single median under-corrects; re-estimate skew per short window (e.g. every 5 minutes) rather than once per batch. Leap seconds are rare but real — clamp any single-step correction to under 2 seconds so a genuine leap second is not mistaken for a monotonicity violation and dropped.
  • Timezone contamination. The most common silent corruption is a tz-naive timestamp that some library interprets as local time, shifting every position by hours. reconstruct_utc raises on tz-naive input for exactly this reason. Once timestamps are trustworthy, the residual duplicates that skew created will collapse cleanly under deduplicating AIS messages in real-time streams, and any remaining temporal holes belong to AIS data quality & gap filling.

Up: Real-Time AIS Stream Ingestion Pipelines