Coastal & Marine Spatial Analysis Pipelines

Sizing Kafka Partitions for AIS Throughput

Partition count is the one Kafka setting that cannot be lowered. It can be raised, and raising it re-keys the hash so every existing key moves partition and per-key ordering breaks across the boundary — which for an AIS topic keyed on MMSI means every vessel’s track has a discontinuity at the moment of the change. Getting it approximately right at the start is therefore worth more than getting most other settings exactly right. This page sizes it, within the real-time ingestion workflow.

Root cause: three separate limits, all met by one number

Partition count is simultaneously the ceiling on consumer parallelism, the unit of ordering, and the unit of rebalance cost. Those three pull in different directions.

Consumers. A consumer group cannot have more active members than the topic has partitions. Extra consumers sit idle. So the partition count is the maximum parallelism the topic will ever support, forever.

Ordering. Messages are ordered within a partition and not across partitions. Keying on MMSI puts each vessel’s messages in one partition, giving per-vessel ordering — which is the guarantee the whole downstream state machine depends on.

Rebalance. Every partition is tracked by the group coordinator, and a rebalance reassigns all of them. With thousands of partitions a rebalance takes tens of seconds, during which the group consumes nothing, and rebalances happen on every deploy, every scale event and every consumer restart.

import math


def partition_floor(peak_msgs_s: float, per_consumer_msgs_s: float,
                    max_consumers: int, skew_factor: float = 3.0) -> int:
    """Minimum partitions from throughput, parallelism and key skew."""
    if min(peak_msgs_s, per_consumer_msgs_s, max_consumers) <= 0:
        raise ValueError("all inputs must be positive")
    by_throughput = math.ceil(peak_msgs_s / per_consumer_msgs_s)
    by_parallelism = max_consumers
    return int(max(by_throughput, by_parallelism) * skew_factor)

The skew factor is the part people leave out, and it is the reason a topic sized exactly at the consumer count performs worse than one sized at three times it.

MMSI Keys Are Not Uniform, So Partitions Are Not Either The upper panel is a ranked plot of message rate per vessel over one hour: a handful of fast ferries and pilot vessels at the left produce around ninety times the median rate, falling steeply to a long flat tail of moored and slow vessels. The lower panel shows the resulting per-partition load in two configurations. With eight partitions and eight consumers, one partition carries three times the load of the lightest and the group's throughput is set by that one consumer. With twenty-four partitions and the same eight consumers, the heavy keys are spread across more partitions and the scheduler can give a consumer holding a heavy partition fewer light ones, so the spread between busiest and quietest consumer narrows substantially. message rate by vessel, ranked 90× median moored and slow vessels 8 partitions, 8 consumers busiest is 3× the lightest 24 partitions, 8 consumers the scheduler can balance heavy against light Extra partitions do not add throughput. They add the scheduler's freedom to even out a skewed key distribution, which on an AIS topic is the difference between the group running at its mean rate and at its slowest member's.

Step-by-step sizing with production code

1. Peak, from the busiest hour you have 

import logging

import pandas as pd

logger = logging.getLogger(__name__)


def peak_rate_msgs_s(counts: pd.Series, *, quantile: float = 0.999) -> float:
    """Peak message rate from a per-second or per-minute count series."""
    if counts.empty:
        raise ValueError("no counts supplied")
    peak = float(counts.quantile(quantile))
    mean = float(counts.mean())
    logger.info("peak %.0f msg/s vs mean %.0f (%.1f× peak-to-mean)", peak, mean, peak / mean)
    if peak / mean > 4.0:
        logger.warning("peak-to-mean above 4× — size on the peak, not on the average")
    return peak

A peak-to-mean ratio of three or four is normal for a coastal AIS feed: traffic follows a daily cycle, fishing fleets sail on a tide, and a single passing cruise ship with a thousand Class B tenders around it produces a burst. Sizing on the mean gives a topic that is comfortable for twenty hours a day and lags for four.

2. Skew, measured rather than assumed 

def key_skew(msgs_per_key: pd.Series) -> float:
    """Ratio of the busiest key's rate to the median key's rate."""
    if msgs_per_key.empty:
        raise ValueError("no per-key counts supplied")
    top = float(msgs_per_key.max())
    med = float(msgs_per_key.median()) or 1.0
    logger.info("key skew: busiest %.0f, median %.0f, ratio %.0f×", top, med, top / med)
    return top / med

An AIS feed typically shows a ratio between 50 and 150. That is not a problem in itself — one busy key is a small fraction of a partition’s capacity — but it means partitions are unevenly loaded, and the group’s throughput is set by its busiest member, not by its average.

3. Check the rebalance side before committing

More partitions is not free. Every rebalance reassigns all of them, and the group is stalled while it happens:

REBALANCE_MS_PER_PARTITION = 4.0      # measure on your cluster; this is a starting point


def rebalance_budget(partitions: int, deploys_per_day: int = 6) -> dict[str, float]:
    """Stall time per rebalance and per day, at this partition count."""
    per_rebalance_s = partitions * REBALANCE_MS_PER_PARTITION / 1000.0
    per_day_s = per_rebalance_s * deploys_per_day
    logger.info("%d partitions: %.1f s per rebalance, %.0f s/day of stalled consumption",
                partitions, per_rebalance_s, per_day_s)
    if per_rebalance_s > 30.0:
        logger.warning("a rebalance stalls the group for over 30 s — consider "
                       "cooperative-sticky assignment or fewer partitions")
    return {"per_rebalance_s": per_rebalance_s, "per_day_s": per_day_s}

For a real-time feed the stall matters directly: consumption stops, the broker keeps receiving, and lag builds at the full ingest rate for the duration. Thirty seconds at 4,000 messages a second is 120,000 messages to catch up on, and the catch-up itself competes with the live feed.

4. Put the three together, and write down why 

def choose_partitions(peak_msgs_s: float, per_consumer_msgs_s: float,
                      max_consumers: int, skew: float) -> dict[str, object]:
    floor = partition_floor(peak_msgs_s, per_consumer_msgs_s, max_consumers,
                            skew_factor=min(max(skew / 30.0, 2.0), 4.0))
    # Round to a power of two so the murmur2 hash distributes evenly.
    n = 1 << (floor - 1).bit_length()
    budget = rebalance_budget(n)
    return {
        "partitions": n,
        "basis": {
            "peak_msgs_s": peak_msgs_s,
            "per_consumer_msgs_s": per_consumer_msgs_s,
            "max_consumers_planned": max_consumers,
            "measured_key_skew": skew,
            "rebalance_s": budget["per_rebalance_s"],
        },
    }

The basis dictionary belongs in the topic’s documentation, not just in a log line. Two years later somebody will ask why the topic has 64 partitions, and the honest answer — “we measured 3,800 msg/s peak, planned for 16 consumers, and saw 90× key skew” — is the difference between a capacity review and an argument.

Two Curves, One Band A chart with partition count on the horizontal axis from 4 to 512. One curve shows the maximum useful consumer count, which rises linearly with partitions and is the reason to want more. A second curve shows the rebalance stall time, which also rises linearly and is the reason to want fewer, crossing a thirty-second acceptability line at about 512 partitions. A shaded band between 32 and 128 marks the region where the parallelism ceiling comfortably exceeds the planned consumer count while rebalances stay under a second, and is annotated as the range to choose from for a regional AIS feed. partitions the usable band consumer ceiling — the reason for more rebalance stall — the reason for fewer 30 s stall — unacceptable for a live feed 432128512 Both curves are linear, so the band is set by the planned consumer count — which is why the basis matters more than the number.

Verification and acceptance test 

import pytest


def test_partition_count_exceeds_planned_consumers(topic_config, capacity_plan):
    assert topic_config["partitions"] >= capacity_plan["max_consumers"], (
        "the topic caps parallelism below the planned consumer count, and this "
        "cannot be fixed without breaking per-MMSI ordering")


def test_partition_count_is_a_power_of_two(topic_config):
    n = topic_config["partitions"]
    assert n & (n - 1) == 0, f"{n} is not a power of two; murmur2 will distribute unevenly"


def test_no_partition_carries_more_than_twice_the_median(partition_rates):
    med = sorted(partition_rates)[len(partition_rates) // 2]
    assert max(partition_rates) <= 2.5 * med, (
        f"partition skew {max(partition_rates) / med:.1f}× — a hot key needs "
        "its own handling, not more partitions")

The third test is the one that catches the case more partitions cannot fix: a single MMSI producing so much traffic that it saturates a partition on its own. That vessel needs a composite key or a dedicated topic, and no amount of repartitioning helps.

What a repartition actually costs, and the alternative

Because the count cannot be lowered and raising it breaks ordering, the realistic options when a topic turns out to be undersized are all migrations. Knowing their shapes in advance is what makes the initial sizing decision feel proportionate.

Raise in place. One command, immediate, and every existing key rehashes. Downstream state machines see each vessel’s track jump partitions at a known instant. Where consumers hold per-vessel state — gap detection, dead reckoning, behaviour segmentation — that state is now on the wrong consumer and must be rebuilt. Viable for stateless consumers, disruptive otherwise.

Drain and switch. Create a new topic at the target size, dual-write, let consumers catch up on the new one, cut over, retire the old. No ordering break, no state loss, and it costs a period of double ingest and a coordinated cutover across every consumer group.

Shard the topic. Split by region or vessel class into several topics, each independently sized. This is the option that scales furthest and the one that changes the most code, because every consumer now subscribes to a set rather than to a name.

def migration_cost(partitions_now: int, partitions_target: int,
                   stateful_consumers: int, retention_h: float) -> dict[str, object]:
    """A rough comparison, so the choice is made on numbers rather than instinct."""
    return {
        "raise_in_place": {
            "downtime_s": 0,
            "state_rebuilds": stateful_consumers,
            "ordering_break": True,
        },
        "drain_and_switch": {
            "double_ingest_h": retention_h,
            "state_rebuilds": 0,
            "ordering_break": False,
        },
        "shard": {
            "code_changes": "every consumer",
            "state_rebuilds": 0,
            "ordering_break": False,
        },
    }

None of the three is cheap, and all of them are cheaper than discovering at peak season that the topic caps parallelism below what the fleet now needs. That asymmetry is the argument for the skew factor in the original sizing: over-provisioning partitions costs a slower rebalance, and under-provisioning costs a migration.

Every Fix Is a Migration; Over-Provisioning Is Not A comparison of three remedies for an undersized Kafka topic against the cost of having over-provisioned it initially. Raising the partition count in place is immediate and breaks per-key ordering, forcing every stateful consumer to rebuild. Draining onto a new topic preserves ordering but requires a period of double ingest and a coordinated cutover across all consumer groups. Sharding into several topics scales furthest and changes code in every consumer. The fourth row, over-provisioning at the outset, costs only a proportionally slower rebalance, which is a fraction of a second per deploy. The asymmetry between the three remedies and the one precaution is the argument for the skew factor in the sizing formula. optionorderingstate rebuildcoordination raise in placebrokenevery consumernone drain and switchpreservednoneall groups, one cutover shard the topicpreservednonecode change everywhere over-provision up frontn/anonea slower rebalance The shaded row costs a fraction of a second per deploy. The other three cost a project. That asymmetry is the whole justification for the skew factor.

Edge cases and gotchas

  • Increasing partitions breaks per-key ordering at the boundary. Existing messages stay where they were; new messages for the same MMSI hash to a different partition. Every consumer state machine sees a track that jumps. Plan a drain-and-switch rather than a live increase, or accept the discontinuity and record its timestamp.
  • cooperative-sticky assignment changes the rebalance arithmetic. It reassigns only the partitions that must move, so the stall is proportional to the change rather than to the topic size. On a large topic it is the difference between a viable partition count and an unusable one.
  • Class B growth is a capacity trend, not noise. Class B carriage rises year on year in most jurisdictions, so a topic sized on today’s message rate will be undersized within a few years by a factor nobody chose. Size against the projection, not the measurement.

Up: Real-Time AIS Stream Ingestion Pipelines