Coastal & Marine Spatial Analysis Pipelines

Filtering Invalid AIS Positions and SOG Outliers

An AIS feed contains three distinct kinds of bad position, and treating them with one statistical filter removes the wrong things. Defined sentinels decode to coordinates in the Arctic Ocean and are not outliers at all — they are the protocol saying “unavailable”. Encoding-range violations are corrupt bits. Kinematically impossible jumps are real reports with a wrong position, and identifying which of two fixes is wrong takes more than a threshold. This page is the three-layer filter, within the AIS data quality workflow.

Root cause: the protocol has “not available” values that look like data

AIS packs position into a fixed-width integer field, and the specification reserves particular values for “not available”. Decoded naively they become coordinates:

Field Sentinel (raw) Decodes to What it means
Longitude 0x6791AC0 181.0° position not available
Latitude 0x3412140 91.0° position not available
Speed over ground 1023 102.3 kn speed not available
Speed over ground 1022 102.2 kn 102.2 kn or higher
Course over ground 3600 360.0° course not available
True heading 511 511° heading not available
Rate of turn −128 — not available

The longitude sentinel is the one that causes the most damage, because 181° is outside the valid range and many decoders wrap it to −179°, producing a position in the Bering Sea. A pipeline with no sentinel check accumulates a dense artificial cluster there, and a fleet-wide analysis finds a “shipping hotspot” in open water.

import numpy as np
import pandas as pd

SENTINELS = {
    "lon": 181.0,
    "lat": 91.0,
    "sog_kn": 102.3,
    "cog_deg": 360.0,
    "heading_deg": 511.0,
}


def sentinel_mask(df: pd.DataFrame) -> pd.Series:
    """True where any field carries its defined not-available value."""
    m = pd.Series(False, index=df.index)
    for col, value in SENTINELS.items():
        if col in df.columns:
            m |= np.isclose(df[col], value, atol=1e-6)
    return m

These are not outliers to be trimmed. They are the vessel telling you it has no fix, and the correct response is to record “no position” rather than to interpolate over the gap as though a report were missing.

Three Layers, Three Different Faults A left-to-right filter chain with three stages, each annotated with what it catches and what it must not catch. Stage one, sentinel rejection, removes the protocol's defined not-available values such as longitude 181 and speed 102.3 knots; a note says these are not outliers and must be recorded as no-position rather than interpolated over. Stage two, range bounds, removes values outside the encoding's legal range, which indicate corrupt bits. Stage three, implied-speed rejection, removes fixes whose distance from the previous fix implies an impossible speed; a note says this stage involves two fixes and must decide which of them is wrong. A footer records that running stage three first would attribute a sentinel jump to a real fix and delete the wrong one. raw fixes from the decoder 1 · sentinels lon 181 · sog 102.3 not errors — protocol values 2 · range bounds |lat| ≤ 90 · |lon| ≤ 180 corrupt bits 3 · implied speed distance ÷ time vs a bound two fixes — which one is wrong? Order matters, and only in one direction. Run stage three first and a sentinel at longitude 181 becomes an impossible jump away from a perfectly good fix — and the attribution rule will blame whichever of the two looks worse, which is a coin flip. Stage one removes it as a protocol value, so stage three never sees the pair and the good fix survives. Each stage writes its own reason code. The three rejection rates behave differently and diagnose different faults: sentinels track receiver quality, range violations track link corruption, jumps track spoofing and multipath.

Step-by-step filter with production code

1. Bounds, from the encoding rather than from geography 

import logging

logger = logging.getLogger(__name__)

BOUNDS = {
    "lat": (-90.0, 90.0),
    "lon": (-180.0, 180.0),
    "sog_kn": (0.0, 102.1),      # 102.2 and 102.3 are sentinels, handled above
    "cog_deg": (0.0, 359.9),
}


def out_of_range_mask(df: pd.DataFrame) -> pd.Series:
    m = pd.Series(False, index=df.index)
    for col, (lo, hi) in BOUNDS.items():
        if col in df.columns:
            bad = (df[col] < lo) | (df[col] > hi)
            if bad.any():
                logger.info("%s: %d value(s) outside [%s, %s]", col, int(bad.sum()), lo, hi)
            m |= bad
    return m

These are encoding limits, not survey limits. Restricting longitude to a study area belongs elsewhere: a vessel legitimately leaving the area is not corrupt data, and conflating the two makes the corruption rate a function of where the study area is drawn.

2. Implied speed, and deciding which fix is wrong

A pair of consecutive fixes implies a speed. When that speed is impossible, one of the two positions is wrong — and naively dropping the second one is right about half the time.

from pyproj import Geod

GEOD = Geod(ellps="WGS84")
MAX_SPEED_KN = {"cargo": 30.0, "tanker": 22.0, "passenger": 45.0, "default": 40.0}


def implied_speed_kn(df: pd.DataFrame) -> pd.Series:
    """Speed implied by the distance and time between consecutive fixes."""
    d = df.sort_values(["mmsi", "timestamp"])
    same = d["mmsi"].to_numpy()[1:] == d["mmsi"].to_numpy()[:-1]
    dt_s = np.diff(d["timestamp"].astype("int64").to_numpy()) / 1e9
    _, _, dist_m = GEOD.inv(d["lon"].to_numpy()[:-1], d["lat"].to_numpy()[:-1],
                            d["lon"].to_numpy()[1:], d["lat"].to_numpy()[1:])
    with np.errstate(divide="ignore", invalid="ignore"):
        kn = (dist_m / np.maximum(dt_s, 1e-9)) * 1.94384
    kn[~same | (dt_s <= 0)] = np.nan
    return pd.Series(np.concatenate([[np.nan], kn]), index=d.index)


def blame_the_outlier(df: pd.DataFrame, limit_kn: float) -> pd.Series:
    """Of two fixes forming an impossible jump, mark the one that is isolated.

    A single bad fix produces two impossible transitions — in and out again.
    A genuine course change produces one. Counting them separates the cases.
    """
    v = implied_speed_kn(df)
    bad_in = v > limit_kn
    bad_out = bad_in.shift(-1, fill_value=False)
    isolated = bad_in & bad_out          # impossible both in and out: this fix is wrong
    logger.info("implied-speed: %d isolated bad fix(es), %d single transitions",
                int(isolated.sum()), int((bad_in & ~bad_out).sum()))
    return isolated

The isolation test is what makes this better than a threshold. One wrong position produces two impossible transitions, one arriving and one leaving; a vessel that genuinely accelerated, or a gap that makes a legitimate move look fast, produces only one. Marking only the doubly-impossible fixes removes the spurious positions and leaves the ambiguous transitions for the gap logic to handle.

3. Cross-check the reported speed

The reported speed over ground is an independent measurement of the same quantity, and disagreement between it and the implied speed is diagnostic:

def speed_disagreement(df: pd.DataFrame) -> pd.Series:
    """Absolute difference between reported and implied speed, in knots."""
    return (df["sog_kn"] - implied_speed_kn(df)).abs()

A vessel reporting 12 knots whose positions imply 600 is almost certainly a position error, because the speed field comes from the same GNSS receiver but a different part of the message and rarely corrupts identically. A vessel reporting 102.2 — the “at or above” sentinel — with an implied speed of 8 is a transponder problem. The two numbers agreeing at an impossible value is the signature of deliberate spoofing rather than corruption, which is where spoofing detection takes over.

One Bad Fix Makes Two Impossible Transitions A vessel track drawn as a sequence of numbered fixes. Between fixes four and five the track jumps far off course to an isolated point and returns at fix six, so both the transition into that point and the transition out of it imply an impossible speed; the point is marked as the wrong fix. Later, between fixes nine and ten, the track makes a sharp but physically possible turn after a long reporting gap, producing a single elevated implied speed; a note says this is ambiguous and belongs to the gap logic rather than to the outlier filter. A summary states that counting impossible transitions per fix — two means the fix is wrong, one means the pair is ambiguous — is what distinguishes them. two impossible transitions → this fix is wrong in out one elevated transition after a long gap — ambiguous Counting impossible transitions per fix: two means the fix is wrong, one means the pair is ambiguous and belongs to the gap logic rather than to this filter. A plain threshold deletes one fix from each case, and is right roughly half the time on the second.

Verification and acceptance test 

import pytest


def test_sentinels_are_not_counted_as_outliers(feed):
    s = sentinel_mask(feed)
    assert s.any(), "the test feed contains no sentinels — the check is unexercised"
    flagged = blame_the_outlier(feed[~s], MAX_SPEED_KN["default"])
    assert not flagged.any(), (
        "an implied-speed outlier survived sentinel removal — the sentinel list is incomplete")


def test_a_single_injected_bad_fix_is_isolated_and_removed(clean_track):
    t = clean_track.copy()
    t.loc[t.index[10], ["lat", "lon"]] = [t.lat.iloc[10] + 3.0, t.lon.iloc[10] + 3.0]
    flagged = blame_the_outlier(t, MAX_SPEED_KN["default"])
    assert flagged.iloc[10], "the injected fix was not detected"
    assert flagged.sum() == 1, f"{flagged.sum()} fixes flagged; only one is wrong"

The second assertion — exactly one — is what keeps the filter from becoming greedy. A threshold-only implementation flags both fixes either side of the injected one and fails it.

Rejection rates are a monitoring signal, not waste

Each of the three layers has its own rate, and the three move for different reasons. Exporting them separately turns the filter from a cleaning step into the pipeline’s earliest quality instrument.

Sentinel rate tracks how many vessels have a GNSS fix. It rises in areas with poor sky view, during jamming, and when a fleet of older transponders enters the area. A step change with no weather or geography behind it usually means a new source was added to the feed.

Range-violation rate tracks link corruption, and on a clean HTTPS-delivered feed it should be essentially zero. Anything above about one in ten thousand means bytes are being mangled between the source and the decoder, and the place to look is the transport rather than the data.

Implied-speed rejection rate tracks the environment: multipath in a port with tall structures, spoofing in contested waters, and GNSS degradation generally. It is the only one of the three that carries operational meaning rather than infrastructure meaning.

def export_filter_rates(counts: dict[str, int], total: int) -> dict[str, float]:
    """Per-layer rejection rates, for the metrics endpoint."""
    if total <= 0:
        raise ValueError("total must be positive")
    rates = {k: v / total for k, v in counts.items()}
    if rates.get("range", 0.0) > 1e-4:
        logger.warning("range-violation rate %.4f%% — suspect the transport, not the feed",
                       rates["range"] * 100)
    return rates

Rolling the three into one “bad data” percentage loses all of that, and it is the shape the metric takes by default because one number is easier to put on a dashboard. The monitoring topic makes the same argument about gate residuals: the aggregate is the number you look at, and the breakdown is the number you act on.

Three Rates, Three Independent Stories Three time series over one week on a shared axis. The sentinel rate is flat near two per cent and rises to nine per cent for one afternoon, annotated as a jamming event in which vessels have no GNSS fix. The range-violation rate is flat at essentially zero and spikes sharply for two hours, annotated as a transport change corrupting bytes between the source and the decoder. The implied-speed rejection rate has a mild daily cycle peaking each afternoon, annotated as multipath in a port with tall structures. A note observes that a single combined bad-data percentage would show one lumpy line in which none of the three events is identifiable. sentinel jamming range transport change implied speed daily multipath cycle in port One combined percentage is a single lumpy line in which none of these three events can be identified.

Edge cases and gotchas

  • A stationary vessel produces zero-distance pairs at high report rates. Dividing by a two-second interval is fine; dividing by a sub-second one after deduplication has merged timestamps is not. Guard the denominator and treat identical timestamps as duplicates rather than as instantaneous motion.
  • Speed 102.2 means “102.2 knots or greater”, not “not available”. Only 102.3 is the unavailable sentinel. Treating 102.2 as missing discards the fastest vessels, which are exactly the ones a traffic study cares about.
  • The implied-speed bound is per vessel class, and the class comes from the static message. Before the first type-5 message arrives, the class is unknown, and using a generous default rather than skipping the check is the safer choice — a missed outlier is recoverable, a deleted fast ferry is not.

Up: AIS Data Quality & Gap Filling