8 min read1,827 words

Deduplicating SCADA and AMI Signals for One Event

A single physical fault announces itself twice. The protective device trips and SCADA reports it within a second; the meters below it lose power and report last gasps over the following half minute. Treated separately, one fault becomes two events, two impact sets and two crew assignments. Merged on timing alone, two genuinely separate faults on adjacent feeders become one event with an impact set spanning both circuits — and the second fault has nobody assigned to it. The correct test is not temporal proximity but explanation: a device trip explains the gasps below it, and “below” is a question for the network graph. This guide builds that correlator, extending the ingestion discipline in real-time telemetry: SCADA and AMI integration.

Environment Prerequisites

  • Python 3.11 with networkx>=3.0 for the reachability test and pandas>=2.0 for windowing.
  • Both feeds resolved to network features before correlation, per the ingestion and last-gasp correlation guides. Signals that have not been resolved cannot be tested for reachability.
  • A cached directed graph oriented source-to-load, refreshed on the topology validation cadence, so the downstream test is a graph query rather than a database round trip.
  • A correlation window sized from measured mesh skew, and a shorter window in storm mode.
  • A dedup key computed after resolution, never from the raw message, so a replay collapses rather than duplicating.
  • An idempotent event store that accepts upserts on that key.

Schema-Aware Validation Protocol — Run Before Correlating

  1. Confirm every signal carries a resolved feature. Correlation on unresolved signals is correlation on coordinates, and coordinates cannot express “downstream of”.
  2. Check the graph is current. A downstream test against a graph older than the last switching operation will merge across a tie that is now open.
  3. Verify event time is populated on both feeds. SCADA and AMI clocks must be comparable, and a feed reporting only arrival time cannot participate in windowing.
  4. Confirm the window is sized from measurement. A window wider than the skew merges distinct events; narrower than the skew splits one.
  5. Test the dedup key on a replay. Feed the same batch twice and assert the event count does not change. If it does, the key is being computed before resolution.

Minimal Reproducible Implementation

from __future__ import annotations

import hashlib
import logging
from dataclasses import dataclass, field
from datetime import datetime, timedelta

import networkx as nx

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("signal-dedup")


@dataclass
class Signal:
    source: str            # SCADA | AMI
    feature_id: str        # resolved network feature
    event_time: datetime
    kind: str              # TRIP | LOCKOUT | LAST_GASP


@dataclass
class Event:
    key: str
    cause_feature: str
    signals: list[Signal] = field(default_factory=list)

    @property
    def evidence(self) -> dict[str, int]:
        return {s.source: sum(1 for x in self.signals if x.source == s.source)
                for s in self.signals}


def _key(cause_feature: str, bucket_start: datetime) -> str:
    raw = f"{cause_feature}|{bucket_start.isoformat()}".encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:16]


def correlate(
    signals: list[Signal],
    graph: nx.DiGraph,
    window: timedelta,
) -> list[Event]:
    """Collapse signals that one device operation explains into single events.

    A device operation explains a last gasp when the gasp's feature is reachable from
    the operated device in the source-to-load graph. Gasps with no operation above them
    form their own event, because a fault on an unmonitored lateral is invisible to
    SCADA and is exactly the case AMI exists to catch.
    """
    ordered = sorted(signals, key=lambda s: s.event_time)
    operations = [s for s in ordered if s.kind in {"TRIP", "LOCKOUT"}]
    events: dict[str, Event] = {}

    for op in operations:
        bucket = op.event_time
        key = _key(op.feature_id, bucket)
        events[key] = Event(key=key, cause_feature=op.feature_id, signals=[op])

    unexplained: list[Signal] = []
    for gasp in (s for s in ordered if s.kind == "LAST_GASP"):
        parent = None
        for op in operations:
            if abs((gasp.event_time - op.event_time)) > window:
                continue
            if gasp.feature_id == op.feature_id or nx.has_path(
                    graph, op.feature_id, gasp.feature_id):
                parent = op
                break
        if parent is None:
            unexplained.append(gasp)
        else:
            events[_key(parent.feature_id, parent.event_time)].signals.append(gasp)

    # Unexplained gasps cluster among themselves, keyed on their common ancestor.
    for gasp in unexplained:
        ancestors = list(nx.ancestors(graph, gasp.feature_id)) or [gasp.feature_id]
        anchor = ancestors[-1]
        bucket = gasp.event_time.replace(second=0, microsecond=0)
        key = _key(anchor, bucket)
        events.setdefault(key, Event(key=key, cause_feature=anchor)).signals.append(gasp)

    LOG.info("%d signal(s) collapsed into %d event(s)", len(signals), len(events))
    return list(events.values())
One physical fault seen by two systems, collapsed into one event A recloser trip arrives from SCADA within a second. Last-gasp messages from meters below it arrive over the following half-minute. Both describe the same fault. The correlator resolves each signal to a network feature, buckets them by event time, and finds that the meters are downstream of the tripped device — which is what justifies collapsing them rather than the fact that they arrived together. SCADA AMI head end Correlator Event store recloser trip, t+0.4 s last-gasp burst, t+2–30 s resolve both to features topology decides, not timing alone one event, two evidence sources Timing suggests the merge; topology is what justifies it.

nx.has_path is doing the work that timing cannot. Two signals inside the same window on unrelated feeders have no path between them and stay separate, which is the behaviour that keeps a storm from producing one enormous event with an impact set covering half the territory.

Should two signals collapse into one event? Two signals merge when one explains the other. A device trip explains every last gasp downstream of it, which is a topology question rather than a timing one. Two trips on unrelated feeders inside the same window are two events no matter how close together they arrived. And a last-gasp burst with no device operation above it is its own event, because something failed that SCADA cannot see. Two signals inside the correlation window one explains the other Is the second downstream of the first? merge Yes — one event, device trip is the cause keep both No — two events on unrelated feeders no device operation Gasps with nothing above them keep Its own event — a failure SCADA cannot see why Common for taps, services and unmonitored laterals Timing narrows the candidates; the graph decides.

Production Deployment Pattern

  1. Resolve before you correlate, always. The whole design depends on both feeds carrying a network feature, and a pipeline that correlates first is correlating coordinates.
  2. Refresh the graph on the validation cadence. A downstream test against a stale graph merges across ties that have since opened.
  3. Shorten the window in storm mode and stamp the window used onto the event, so a post-event review can see which parameters produced the grouping.
  4. Treat unexplained gasps as first-class events. They are the faults SCADA cannot see — service and lateral failures — and suppressing them because no device operated loses the only evidence of them.
  5. Assert idempotency in the deployment pipeline. A replay test that asserts a stable event count catches the key-before-resolution defect before production does.
  6. Publish the evidence mix per event. An event supported by a device trip and two hundred gasps is more certain than one supported by three gasps, and downstream consumers should be able to see the difference.
  7. Measure the merge rate. The ratio of raw signals to events is stable for a given estate, and a sudden move in it — in either direction — usually means the window or the graph changed rather than the weather, and both of those are worth knowing before the next event rather than after it.
What each deduplication mistake costs downstream Over-merging and under-merging both produce a wrong picture, and they fail in opposite directions. Over-merging hides a second fault inside the first one. Under-merging inflates the event count, splits the impact set and sends two crews to one problem. Failure Cause Cost Over-merged window too wide second fault hidden Under-merged window too narrow two crews, one fault Merged across feeders no topology check impact set spans two circuits Duplicate on replay key built before resolution event count inflated Over-merging is the dangerous direction: a hidden fault has nobody assigned to it.

Conclusion

Deduplication that merges on timing produces a picture that is confidently wrong during exactly the conditions it matters. Requiring one signal to explain the other — a device operation with the gasps reachable below it — makes the merge a statement about the network rather than about the clock. Keeping unexplained gasps as their own events preserves the faults SCADA cannot see, and keying events after resolution makes replay harmless. What reaches impact analysis afterwards is one event per fault, with the evidence that supports it attached.

For authoritative reference, consult the NetworkX shortest-path documentation and the Python hashlib module.

Related pages