9 min read1,984 words

Storm-Mode Thresholds for Outage Detection

Outage detection is a signal-to-noise problem, and every threshold in it encodes an assumption about the noise. A corroboration threshold of three last-gasp messages inside a window is excellent evidence on a quiet evening, when a feeder produces almost no traffic. During a wind event, the same feeder produces intermittent single-meter drops continuously, and three messages is a coincidence. Nothing in the detection stack notices the difference: it confirms events at the same rate it always did, and the operators discover that the system is reporting more outages than the field has. This guide replaces the fixed count with a threshold computed against each feeder’s own recent baseline, adds the absolute floor that keeps quiet feeders from confirming on noise, and stamps the thresholds in force onto every confirmation so a post-event review can tell which standard an event met. It is the detection half of the regime model described in storm response and restoration analytics.

Environment Prerequisites

  • Python 3.11 in an isolated environment with pandas>=2.0 for the windowing, and no dependency on a geodatabase — this layer works on the signal stream, not the network.
  • A last-gasp and SCADA feed with event time and arrival time kept distinct, as produced by the telemetry ingestion pipeline.
  • A per-feeder baseline store — a rolling window of message rates, persisted so a restart does not begin with an empty baseline and confirm everything.
  • The measured mesh skew for the window size, and a shortened storm window derived from it rather than guessed.
  • Regime state available to the detector, so it knows which parameter set is in force. A detector that infers the regime from its own inputs will flap between them.
  • An event store that accepts the threshold stamp — the baseline, ratio and floor in force — alongside each confirmation.

Schema-Aware Validation Protocol — Run Before Trusting a Threshold

  1. Confirm the baseline window is long enough to be stable. A baseline computed over two minutes tracks the storm itself and cancels out the signal you are trying to detect. Minutes to tens of minutes is the usual band; measure it against a past event.
  2. Confirm every feeder has a baseline. A feeder with no history defaults to zero, and any traffic at all then looks like an infinite ratio. New feeders must inherit a conservative default until they have their own.
  3. Verify the absolute floor against the smallest real feeder. The floor exists to stop a two-meter lateral from confirming on two messages, so it has to be set from the smallest feeder in the estate, not from the average.
  4. Check event time is populated and monotonic. The whole comparison is rate over time; a feed that reports only arrival time makes both the baseline and the bucket meaningless.
  5. Test the regime switch on replayed data. Replay a past storm through both parameter sets and compare confirmations against what actually happened. This is the only calibration that is worth anything, and the data already exists.

Minimal Reproducible Implementation

The detector below maintains a rolling baseline per feeder, evaluates each bucket against both a ratio and an absolute excess, and returns confirmations stamped with the thresholds that produced them. It has no dependency on the network model, so it can run in a container alongside the message broker.

from __future__ import annotations

import logging
from collections import deque
from dataclasses import dataclass, field

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("storm-thresholds")


@dataclass(frozen=True)
class Regime:
    """The parameter set in force. Storm mode shortens the window and leans on ratio."""

    name: str
    window_s: int
    min_ratio: float
    min_excess: float          # messages per minute above baseline
    baseline_windows: int


NORMAL = Regime("NORMAL", window_s=45, min_ratio=3.0, min_excess=1.0, baseline_windows=20)
ELEVATED = Regime("ELEVATED", window_s=45, min_ratio=4.0, min_excess=2.5, baseline_windows=12)
STORM = Regime("STORM", window_s=20, min_ratio=5.0, min_excess=6.0, baseline_windows=8)


@dataclass
class Confirmation:
    feeder: str
    observed_rate: float
    baseline_rate: float
    regime: str
    ratio: float
    excess: float


@dataclass
class Detector:
    regime: Regime = NORMAL
    _history: dict[str, deque] = field(default_factory=dict)

    def _baseline(self, feeder: str) -> float:
        """Trailing mean rate for a feeder, in messages per minute.

        A feeder with no history returns a conservative default rather than zero, because
        a zero baseline makes every ratio infinite and confirms on the first message.
        """
        hist = self._history.get(feeder)
        if not hist:
            return 0.5
        return sum(hist) / len(hist)

    def observe(self, feeder: str, count: int) -> Confirmation | None:
        """Feed one window's message count for a feeder; return a confirmation or None."""
        rate = count / (self.regime.window_s / 60.0)
        base = self._baseline(feeder)

        hist = self._history.setdefault(
            feeder, deque(maxlen=self.regime.baseline_windows))

        ratio = rate / base if base > 0 else float("inf")
        excess = rate - base
        confirmed = ratio >= self.regime.min_ratio and excess >= self.regime.min_excess

        # The baseline must not absorb the outage it is meant to detect.
        if not confirmed:
            hist.append(rate)
            return None

        LOG.info("%s confirmed: rate=%.1f base=%.1f ratio=%.1f regime=%s",
                 feeder, rate, base, ratio, self.regime.name)
        return Confirmation(feeder=feeder, observed_rate=rate, baseline_rate=base,
                            regime=self.regime.name, ratio=ratio, excess=excess)

    def set_regime(self, regime: Regime) -> None:
        """Change parameter set. Histories are trimmed, never cleared."""
        self.regime = regime
        for feeder, hist in self._history.items():
            self._history[feeder] = deque(
                list(hist)[-regime.baseline_windows:], maxlen=regime.baseline_windows)
        LOG.info("regime set to %s", regime.name)
Why an absolute corroboration threshold stops working as the arrival rate rises On a quiet evening a feeder produces almost no last-gasp traffic, so three corroborating messages inside the window is overwhelming evidence of a real outage. During a wind event the same feeder produces intermittent single-meter drops continuously, and three messages is a coincidence rather than evidence. The threshold that discriminated at the first rate confirms noise at the last, which is how a storm produces more outages in the system than in the field. baseline last-gasp traffic per feeder Quiet evening 0.4 msg/min 3 is strong evidence Windy afternoon 2.6 msg/min 3 is weak evidence Active storm 14.0 msg/min 3 is a coincidence The evidence a count represents depends entirely on the baseline underneath it.

The line worth noticing is the one that appends to the history only when the window did not confirm. A baseline that absorbs the outage it is supposed to detect will, after a few windows, decide that the elevated rate is normal and stop confirming — which is exactly the failure mode a naive rolling mean produces during a long event.

A relative threshold computed from the feeder’s own recent baseline Each cycle measures the trailing baseline rate per feeder over a window long enough to be stable and short enough to track the weather. The observed rate in the current bucket is compared against that baseline rather than against a fixed number. A feeder is promoted to confirmed only when the excess is both large in ratio and large in absolute terms, which is what stops a rural lateral with a baseline near zero from confirming on two messages. The thresholds in force are stamped onto the event so a later review can see what standard it met. BASELINE trailing rate per feeder OBSERVE bucket rate in the window COMPARE ratio AND absolute excess CONFIRM stamp the thresholds used ratio alone is not enough RURAL LATERAL GUARD a near-zero baseline makes any ratio large — the absolute floor is what saves it Ratio catches the storm case; the absolute floor catches the quiet-feeder case.

Production Deployment Pattern

  1. Persist the baselines. A detector restarted mid-storm with empty histories will confirm everything for the first several windows. Write the histories to a store the process can recover from.
  2. Drive the regime from the operational declaration, not from the detector’s own inputs. A detector that promotes itself to storm mode on its own confirmations has a feedback loop.
  3. Stamp every confirmation. The regime, baseline, ratio and floor belong on the event. Without them, a post-event review cannot distinguish a genuine confirmation from one produced by a threshold that was wrong for the conditions.
  4. Replay before changing a parameter. Every threshold change should be tested against at least two past events — one quiet, one severe — and the comparison recorded alongside the change.
  5. Alarm on confirmation rate, not on message rate. Message rate rises in every storm; a confirmation rate that rises disproportionately means the thresholds are letting noise through.
  6. Keep the detector independent of the network model. It should be deployable, testable and restartable without a geodatabase, which is what makes replay cheap enough to actually do.
Threshold behaviour across the three regimes, with the failure each setting prevents In normal operation an absolute count is correct and simple. In elevated operation the ratio test takes over while the absolute floor stays, which prevents both phantom confirmations and the rural-lateral false positive. In storm mode the window itself shortens, because a long window during a fast-moving event merges distinct outages on the same feeder into one, and the confirmation is stamped with the thresholds so the post-event review can separate the regimes. Regime Window Test Prevents Normal measured skew absolute count single-meter noise Elevated measured skew ratio + absolute floor phantom confirmations Storm shortened ratio + floor, stamped distinct outages merging Shortening the window is what keeps two outages on one feeder from becoming one.

Conclusion

A fixed corroboration threshold encodes an assumption about the quiet case and applies it to the loud one. Computing the threshold against each feeder’s own recent baseline, guarding it with an absolute floor, shortening the window when events move fast, and stamping the parameters onto every confirmation turns detection into something that behaves consistently across regimes and can be audited afterwards. The confirmations this produces feed impact analysis, and the next question is how long each of those outages will take to restore — which is the subject of estimating restoration times.

For authoritative reference, consult the pandas time-series documentation and the Python collections library.

Related pages