AIS Data Quality & Gap Filling

AIS data quality control is the correctness gate that sits between raw ingestion and every analytical stage of the AIS Vessel Tracking & Route Automation domain. Automatic Identification System position reports arrive already corrupted in predictable ways: a receiver decodes a garbled longitude into an ocean-crossing teleport, two shore stations report the same fix twice, a mis-keyed MMSI merges two vessels into one impossible track, and a satellite pass delivers positions minutes after a terrestrial receiver already logged them. If these defects are not caught and repaired per vessel, they propagate silently — a single spurious 900-knot spike will shatter a clean transit into fragments during segmentation, and a duplicated report will inflate a dwell-time statistic that a compliance analyst later trusts. This page defines the quality dimensions, gives a reference table of QC thresholds, and builds a production pandas pipeline that flags and repairs per-MMSI tracks with typed functions, structured logging, and hard failures rather than silent fallbacks.

AIS Data-Quality Dimensions #

AIS quality problems fall into a small number of recurring classes, and a robust pipeline handles each one explicitly rather than relying on a single downstream outlier filter to absorb them all.

Positional outliers. An individual fix lands hundreds of kilometres from its neighbours, then the track returns to a plausible position on the next report. These “teleport” spikes come from bit errors in the 168-bit AIS position payload that survive the CRC in some receiver firmware, or from coordinate-field byte swaps in shore-side aggregation.

Implausible speeds and jumps. The distance between two consecutive fixes, divided by the elapsed time, implies a speed over ground (SOG) no real vessel can sustain. A container ship tops out near 25 knots and a fast ferry near 45; an implied 300-knot leg is definitionally a data error, either in a position or in a timestamp.

Duplicate reports. The same transmission is received and logged by two overlapping terrestrial receivers, or by a terrestrial receiver and a satellite. Exact duplicates (identical MMSI, timestamp, and coordinates) are harmless once removed, but near-duplicates with slightly different receiver timestamps require a tolerance-based key.

MMSI errors. The Maritime Mobile Service Identity is the vessel primary key, and it is frequently wrong: a factory-default 000000000, a nine-digit value outside assigned ranges, or a shared MMSI where two vessels transmit the same identity, producing a single track that jumps between two locations.

Coverage gaps from VHF/satellite handoff. A vessel sails out of terrestrial VHF range (roughly 40 nautical miles) and is only seen again on a satellite revisit tens of minutes later. The reconstructed track shows a long straight chord where curved sailing actually occurred — a gap to be flagged and, where appropriate, filled by interpolating gaps in AIS vessel trajectories.

Reference QC Thresholds #

The table below is the single source of truth for the numeric gates the pipeline enforces. Treat the defaults as a starting point calibrated for mixed coastal traffic and tighten MAX_SOG_KN per vessel class where the vessel type is known.

The relationship between the speed and step gates is deliberate. The maximum plausible step distance between two fixes separated by $\Delta t$ seconds is

$$d_{\max} = v_{\max} \cdot \Delta t, \qquad v_{\max} = \frac{\text{MAX\_SOG\_KN} \times 1852}{3600}\ \text{m/s}$$

so the same physical constant drives both the per-leg jump test and the aggregate speed test. A fix that passes the speed gate cannot fail the step gate for the same MAX_SOG_KN; the two exist because timestamps and positions fail independently, and testing both isolates which field is corrupt.

QC Pipeline: Raw to Clean Track #

The diagram traces a raw per-MMSI track through the four ordered stages the code implements: normalize and deduplicate, validate identity and time, flag kinematic outliers, then repair into a clean track carrying provenance flags.

The implementation groups reports by MMSI and processes each vessel track independently. Working per vessel is what makes the kinematic tests meaningful: speed and acceleration are only defined along a single vessel’s ordered time axis.

from __future__ import annotations

import logging
from dataclasses import dataclass

import numpy as np
import pandas as pd
from pyproj import Geod

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",
)
logger = logging.getLogger(__name__)

_GEOD = Geod(ellps="WGS84")


@dataclass(frozen=True)
class QCThresholds:
    """Numeric gates for AIS quality control; see the reference table."""

    max_sog_kn: float = 60.0
    max_accel_kn_per_s: float = 1.0
    min_interval_s: float = 1.0
    gap_threshold_s: float = 600.0
    dedup_tol_s: float = 2.0
    mmsi_min: int = 100_000_000
    mmsi_max: int = 799_999_999

    @property
    def max_step_m_per_s(self) -> float:
        """Maximum plausible geodesic step per second, in metres."""
        return self.max_sog_kn * 1852.0 / 3600.0


_REQUIRED_COLUMNS: frozenset[str] = frozenset(
    {"mmsi", "timestamp", "lat", "lon"}
)


def _validate_frame(df: pd.DataFrame) -> None:
    """Raise if the input frame is missing columns or has a bad dtype."""
    missing = _REQUIRED_COLUMNS - set(df.columns)
    if missing:
        raise ValueError(f"AIS frame missing required columns: {sorted(missing)}")
    if not pd.api.types.is_datetime64_any_dtype(df["timestamp"]):
        raise TypeError(
            "Column 'timestamp' must be datetime64; parse it before QC "
            "(pd.to_datetime(..., utc=True))."
        )


def _geodesic_step_m(lat: np.ndarray, lon: np.ndarray) -> np.ndarray:
    """Great-circle distance in metres between consecutive fixes.

    Returns an array aligned to the input; element 0 is NaN because a
    step is undefined for the first fix in a track.
    """
    if lat.size < 2:
        return np.full(lat.size, np.nan, dtype="float64")
    _, _, dist = _GEOD.inv(lon[:-1], lat[:-1], lon[1:], lat[1:])
    return np.concatenate([[np.nan], dist]).astype("float64")


def qc_vessel_track(track: pd.DataFrame, thr: QCThresholds) -> pd.DataFrame:
    """Flag and repair a single-MMSI, time-sorted AIS track.

    Adds boolean columns ``qc_duplicate``, ``qc_speed``, ``qc_gap`` and a
    consolidated ``qc_flag`` string. Rows failing the speed gate are dropped
    (teleport spikes); gap rows are retained and marked for interpolation.

    Raises:
        ValueError: if the track spans more than one MMSI.
    """
    if track["mmsi"].nunique() != 1:
        raise ValueError("qc_vessel_track expects exactly one MMSI per call")

    t = track.sort_values("timestamp").reset_index(drop=True)
    dt_s = t["timestamp"].diff().dt.total_seconds().to_numpy()

    # Near-duplicate: same position within the dedup time tolerance.
    same_pos = (
        t["lat"].round(5).eq(t["lat"].round(5).shift())
        & t["lon"].round(5).eq(t["lon"].round(5).shift())
    ).to_numpy()
    dup = same_pos & (dt_s <= thr.dedup_tol_s)
    dup[0] = False

    step_m = _geodesic_step_m(t["lat"].to_numpy(), t["lon"].to_numpy())
    with np.errstate(invalid="ignore", divide="ignore"):
        sog_ms = step_m / dt_s
        sog_kn = sog_ms * 3600.0 / 1852.0
        accel = np.abs(np.diff(sog_kn, prepend=np.nan)) / dt_s

    speed_bad = (sog_kn > thr.max_sog_kn) | (accel > thr.max_accel_kn_per_s)
    speed_bad = np.nan_to_num(speed_bad, nan=0.0).astype(bool)

    gap = np.nan_to_num(dt_s, nan=0.0) > thr.gap_threshold_s

    t["qc_duplicate"] = dup
    t["qc_speed"] = speed_bad
    t["qc_gap"] = gap
    t["qc_flag"] = np.select(
        [dup, speed_bad, gap],
        ["duplicate", "speed_outlier", "gap"],
        default="ok",
    )

    n_drop = int(dup.sum() + speed_bad.sum())
    logger.info(
        "MMSI %d: %d fixes, dropped %d (dup=%d speed=%d), %d gaps marked",
        int(t["mmsi"].iloc[0]), len(t), n_drop,
        int(dup.sum()), int(speed_bad.sum()), int(gap.sum()),
    )

    keep = ~(t["qc_duplicate"] | t["qc_speed"])
    return t.loc[keep].reset_index(drop=True)


def qc_ais_frame(df: pd.DataFrame, thr: QCThresholds | None = None) -> pd.DataFrame:
    """Apply per-MMSI quality control across a multi-vessel frame."""
    thr = thr or QCThresholds()
    _validate_frame(df)

    valid_mmsi = df["mmsi"].between(thr.mmsi_min, thr.mmsi_max)
    n_bad_id = int((~valid_mmsi).sum())
    if n_bad_id:
        logger.warning("Dropping %d reports with out-of-range MMSI", n_bad_id)
    df = df.loc[valid_mmsi]

    in_range = (
        df["lat"].between(-90.0, 90.0) & df["lon"].between(-180.0, 180.0)
    )
    df = df.loc[in_range]
    if df.empty:
        raise RuntimeError("No AIS reports survived identity/bounds validation")

    cleaned = [qc_vessel_track(g, thr) for _, g in df.groupby("mmsi", sort=False)]
    result = pd.concat(cleaned, ignore_index=True)
    logger.info("QC complete: %d clean fixes across %d vessels",
                len(result), result["mmsi"].nunique())
    return result

Validation Gates and Quality Control #

Three gates guard the pipeline, and each one raises rather than silently continuing so a broken upstream feed surfaces immediately instead of poisoning the archive.

1. Per-MMSI monotonic time. After sorting, the per-vessel timestamp axis must be strictly increasing. A repeated or backwards timestamp means either a duplicate (handled by the dedup key) or receiver-time reordering from satellite latency, which corrupts every step and acceleration computed across it.

2. Speed sanity. No retained fix may imply an SOG above MAX_SOG_KN or an acceleration above MAX_ACCEL_KN_PER_S. These two independent tests localize the fault: a lone position error trips the speed gate on one leg and clears on the next, whereas a timestamp error trips it symmetrically on both adjacent legs.

3. Deduplication key. The (mmsi, timestamp_rounded, lat_rounded, lon_rounded) tuple with a DEDUP_TOL_S tolerance is the canonical identity of a fix. Exact-key collapse is safe; the tolerance handles the same transmission arriving from two receivers with slightly skewed clocks.

def assert_qc_invariants(track: pd.DataFrame, thr: QCThresholds) -> None:
    """Post-QC acceptance gate for a single cleaned vessel track.

    Raises:
        AssertionError: if any invariant the pipeline promises is violated.
    """
    ts = track["timestamp"]
    if not ts.is_monotonic_increasing:
        raise AssertionError("Cleaned track timestamps are not monotonic")

    if len(track) >= 2:
        step = _geodesic_step_m(track["lat"].to_numpy(), track["lon"].to_numpy())
        dt = track["timestamp"].diff().dt.total_seconds().to_numpy()
        # Ignore legs across marked gaps; those are interpolation candidates.
        legs = (dt > 0) & (dt <= thr.gap_threshold_s)
        with np.errstate(invalid="ignore"):
            sog_kn = (step / dt) * 3600.0 / 1852.0
        offenders = np.nan_to_num(sog_kn[legs], nan=0.0) > thr.max_sog_kn
        if offenders.any():
            raise AssertionError(
                f"{int(offenders.sum())} legs still exceed {thr.max_sog_kn} kn"
            )
    logger.info("QC invariants hold for MMSI %d", int(track["mmsi"].iloc[0]))

Run assert_qc_invariants on a sample of cleaned tracks in CI. A green run guarantees the frame handed to segmentation contains no teleport legs and no duplicate-driven zero-duration steps.

Common Failure Modes and Diagnosis #

Satellite-AIS Latency Reordering #

Symptom: A track shows a fix “in the future” relative to its neighbours; steps and accelerations spike then reverse across two adjacent legs with no real manoeuvre.

Diagnosis: Satellite receivers buffer and downlink AIS on each orbital pass, so a message transmitted at 12:03 can be ingested after a terrestrial message transmitted at 12:05. If you sort by ingestion time rather than transmission time, the track zig-zags. Sort strictly on the decoded AIS transmission timestamp, and treat any residual non-monotonicity as clock skew rather than motion. The full receiver-time versus transmit-time treatment lives in Handling AIS Timestamp Drift & Clock Skew.

MMSI Collisions and Shared MMSI #

Symptom: One MMSI produces a track that teleports repeatedly between two distant regions, and every other fix trips the speed gate.

Diagnosis: Two physical vessels are transmitting the same MMSI — a mis-programmed transponder, a cloned identity, or a factory default. The speed gate will shred the interleaved track into fragments but never converge. Detect it by clustering the fixes for a single MMSI spatially: a genuine vessel forms one moving cluster, a collision forms two persistent ones. Split the MMSI into synthetic sub-tracks before QC, or route it to anomaly detection for identity analysis.

Timestamp vs Receiver-Time Skew #

Symptom: Every vessel seen through one receiver has a small constant SOG bias, or dwell times computed at a berth are systematically short.

Diagnosis: The receiver stamps arrival time instead of using the AIS message’s own time field, and its NTP clock has drifted. A constant offset does not change step distances but does distort durations, biasing every speed derived from that feed. Reconcile against a second receiver’s timestamps for the same MMSI and the same fix, and correct the offset before computing kinematics — see the clock-skew page linked above.

Coordinate Field Byte Swap #

Symptom: Isolated fixes appear at mirrored or transposed coordinates (latitude and longitude swapped, or sign flipped) then the track resumes normally.

Diagnosis: A decoding bug in the NMEA sentence parser or an aggregator serialization fault swapped the two-complement coordinate fields. These pass the bounds check when both values are individually valid, so they only surface at the speed gate as a single-fix spike. Diagnose by checking whether the offending fix becomes plausible when lat and lon are swapped; if so, fix the parser rather than discarding data.

Pipeline Integration and Downstream Handoff #

The cleaned frame is the contract every downstream stage depends on. Segmenting vessel routes by behavior and its DBSCAN clustering stage assume the time axis is monotonic and free of teleport spikes; a single un-repaired 900-knot leg would open a spurious segment boundary and fabricate a behavioral primitive that never happened. Speed & heading profiling for maritime analytics computes velocity directly from consecutive fixes, so it inherits every duplicate and skew defect this stage removes — deduplication in particular is load-bearing because a zero-duration duplicated pair produces a divide-by-zero speed.

The QC provenance column travels with the data. Rather than silently deleting flagged fixes, the pipeline records why each one was dropped or marked, so anomaly detection in AIS trajectories can distinguish a benign coverage gap from a suspicious one. That distinction matters: a long gap that QC marks for benign interpolation might, in a different context, be a vessel deliberately going dark — a signal the spoofing and jamming detectors must not have already smoothed away. Persist the flags to Parquet alongside the coordinates so every consumer sees the same provenance.

For the mechanics of filling the gaps this stage marks — geodesic interpolation, max-gap policy, and land-avoidance — continue to interpolating gaps in AIS vessel trajectories. Upstream, the raw records arrive from real-time AIS stream ingestion pipelines, whose bounds and message-type gates catch the grossest corruption before this per-vessel stage runs. Message field semantics and MMSI category prefixes are specified in the ITU-R M.1371 AIS standard and the IMO AIS technical pages.

---

Related #

• Interpolating Gaps in AIS Vessel Trajectories — filling the coverage gaps this stage marks with geodesic and PCHIP methods
• Real-Time AIS Stream Ingestion Pipelines — upstream Kafka ingestion that delivers the raw records QC repairs
• Anomaly Detection in AIS Trajectories — consumes the QC flags to separate benign gaps from spoofing and dark-ship events
• Speed & Heading Profiling for Maritime Analytics — velocity derivation that inherits the dedup and skew fixes made here
• Parsing AIS NMEA Sentences with Python — upstream decoding where coordinate byte-swap defects originate

Up: AIS Vessel Tracking & Route Automation