Real-Time Telemetry: SCADA & AMI Integration for Outage Automation
The Failure Mode: Raw Signals That Never Become Features
The failure this reference solves is the gap between a signal and a feature. A SCADA breaker reports that it tripped, a thousand smart meters emit a dying gasp, and a field fault-indicator flags a passing fault — but none of those payloads names a network feature the trace engine can reason about. Until each raw signal is resolved to a specific junction, span, or service point on the validated graph, no impact set can be computed, no isolation boundary can be drawn, and no crew can be routed. Teams that skip this resolution layer end up correlating outages by hand during exactly the moments — storm peaks — when there is no human to spare, and the result is a detection process that is simultaneously too slow and quietly wrong.
The specific defects are subtle because they pass every surface check. A last-gasp coordinate that snaps to the wrong feeder because the match tolerance is too loose puts real customers on the wrong circuit. A breaker trip and the meter gasps it caused, arriving seconds apart with clock skew between the SCADA historian and the AMI head-end, get treated as separate events and spawn a phantom swarm of tiny outages. A duplicate alarm burst — the same trip re-reported three times as the communications link flaps — inflates the outage into three events that dispatch three crews to one fault. Each of these is a data-integration bug, not a mapping bug, and each corrupts every automated decision downstream. This work sits directly beneath the outage routing and impact automation reference: it is the ingestion layer that produces the feature-resolved fault events every impact and dispatch computation consumes.
Prerequisite Checklist
Confirm every item below before wiring a telemetry feed to production outage detection. Skipping the snapshot or tolerance checks is the most common cause of an ingestion path that reports healthy while attaching signals to the wrong features.
Core Data Model: The Canonical Event Envelope and the Dedup Key
Every telemetry integration begins with one decision: what a normalized fault event looks like before it touches the network. Heterogeneous sources cannot be correlated in their native shapes — a SCADA point-change record, an AMI last-gasp datagram, and a fault-indicator poll response share nothing structurally. The canonical envelope is the shared vocabulary that makes them comparable. It carries the source system, the originating identifier (breaker id, meter id, or indicator id), two timestamps kept strictly separate (event time, when the physical event occurred; arrival time, when the message reached the consumer), the raw reported location, and a payload-specific status. Nothing downstream reads the native format; the envelope is the contract.
Two design commitments make the envelope trustworthy. The first is that event time and arrival time never collapse into one field. Correlation, deduplication, and lifecycle timestamps all key on event time, while arrival time exists only to measure latency and to detect replays; conflating them is the root of phantom-outage swarms. The second is the deduplication key: a deterministic hash of the resolved feature identifier and the correlation window bucket. Because the key is computed from the outage’s identity rather than from the message, three re-reports of one breaker trip produce one key and therefore one event. This idempotency is what lets the consumer be safely restarted, replayed, and scaled horizontally without corrupting outage state.
from __future__ import annotations
import hashlib
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional
@dataclass(frozen=True)
class TelemetryEnvelope:
"""Canonical, source-agnostic representation of one telemetry signal.
Every SCADA, AMI, and fault-indicator payload is normalized into this
shape before correlation or feature resolution runs.
"""
source: str # "SCADA" | "AMI" | "FIELD_FI"
origin_id: str # breaker id, meter id, or indicator id
event_time: datetime # when the physical event occurred (UTC)
arrival_time: datetime # when the message reached the consumer (UTC)
raw_x: float # reported location in the network CRS
raw_y: float
status: str # source-specific status token
feature_id: Optional[str] = field(default=None) # set by resolution
def window_bucket(self, window_seconds: int) -> int:
"""Floor event_time into a fixed correlation-window bucket."""
epoch = self.event_time.astimezone(timezone.utc).timestamp()
return int(epoch // window_seconds)
def dedup_key(self, window_seconds: int) -> str:
"""Deterministic identity: resolved feature + event-time bucket.
Requires feature_id to be set; raises before resolution so a caller
cannot accidentally deduplicate on an unresolved signal.
"""
if self.feature_id is None:
raise ValueError("cannot compute dedup_key before feature resolution")
bucket = self.window_bucket(window_seconds)
raw = f"{self.feature_id}|{bucket}"
return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:24]The status token deserves its own discipline. SCADA reports device operations (TRIP, LOCKOUT, RECLOSE), AMI reports meter conditions (LAST_GASP, POWER_UP), and indicators report FAULT_PASSED. Keeping these as a constrained vocabulary per source — rather than free text — is what lets the correlation and detection logic branch deterministically, exactly as the barrier logic in valve and isolator mapping branches only on a constrained operational-status domain. A single unrecognized token silently drops a signal from a correlation group and shrinks the impact set.
Step-by-Step Implementation
Follow this sequence to take heterogeneous telemetry from raw arrival to a feature-resolved, idempotent event ready for detection.
- Normalize into the envelope. Map each source payload onto
TelemetryEnvelope, keeping event time and arrival time distinct and rejecting any message missing a mandatory field. - Correlate on event time. Bucket envelopes into the correlation window keyed on feeder or SCADA zone, so signals from one physical event cluster together regardless of arrival order.
- Resolve signal to feature. Snap each envelope’s raw location to the nearest service point, device, or span within the network XY tolerance, rejecting matches beyond design accuracy rather than forcing a wrong association.
- Construct the idempotent event. Compute the dedup key from the resolved feature and window bucket, and upsert so duplicates collapse.
- Stage behind a spatial index. Write the event to the staging table so impact and isolation traces read one consistent snapshot.
- Hand off to detection. Emit the staged event to the detection queue, where corroboration promotes it from provisional to confirmed.
The correlation window is the least intuitive step, so it earns a diagram. Event-time windowing groups signals by when the physical event happened, tolerating the reality that messages arrive late, out of order, and in bursts. The window must be wide enough to absorb the worst-case clock skew and communication latency, yet narrow enough that two genuinely distinct outages on the same feeder are not merged. Anchoring on arrival time instead would shatter a single breaker trip and its trailing last-gasp cascade into a scatter of unrelated events.
The resolution step is where spatial precision becomes correctness. Each envelope carries a raw location, and the job is to attach it to exactly one feature — a service point for an AMI gasp, a device for a SCADA trip, a span for a fault-indicator — using the same snapping tolerance the network enforces for junction-edge connectivity. Snapping too loosely attaches a gasp to a neighboring feeder and misstates which customers are out; snapping too tightly leaves the signal orphaned and drops the customer from the impact set entirely. The resolver below buffers candidate features to the tolerance and takes the nearest match, refusing anything beyond the band.
import geopandas as gpd
import logging
from shapely.geometry import Point
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
def resolve_to_feature(
envelope_x: float,
envelope_y: float,
features: gpd.GeoDataFrame,
id_column: str,
tolerance_m: float = 0.25,
) -> str | None:
"""Snap a raw telemetry location to the nearest feature within tolerance.
Returns the feature id, or None when no candidate falls inside the design
accuracy band — an explicit orphan the caller must route, never a forced
wrong association.
"""
if features.crs is None:
raise ValueError("feature layer has no defined CRS; align before resolving")
signal = Point(envelope_x, envelope_y)
# Distance in the projected network CRS; features must share that CRS.
distances = features.geometry.distance(signal)
nearest_idx = distances.idxmin()
nearest_dist = float(distances.loc[nearest_idx])
if nearest_dist > tolerance_m:
logging.warning(
"signal at (%.3f, %.3f) orphaned: nearest feature %.3f m > %.3f m tolerance",
envelope_x, envelope_y, nearest_dist, tolerance_m,
)
return None
feature_id = str(features.loc[nearest_idx, id_column])
logging.info("resolved signal to feature %s at %.3f m", feature_id, nearest_dist)
return feature_idOnce resolution and correlation are in place, the two child procedures make the pattern concrete for each source. Ingesting SCADA fault events with Python works the queue-consumer-to-idempotent-event path end to end for breaker trips and fault-indicators, and correlating AMI last-gasp messages to network features handles the high-volume, clock-skewed meter case where windowing and snapping matter most. The high-throughput staging, transformation, and validation that feed both are the same discipline documented in data ingestion pipelines for utility assets, applied to a live event stream rather than a nightly batch.
Diagnostic Protocol
When outage detection produces the wrong picture, the fault is usually in ingestion, not the trace. Work this checklist in order; the first item is the most common root cause.
- Clock skew masquerading as multiple outages. Compare event time against arrival time on the signals in question. If a single physical event scatters across event-time buckets, the correlation window is narrower than the real skew budget between the SCADA historian and the AMI head-end. Widen the window to the measured worst case before touching anything else.
- Out-of-domain status tokens. Query the staging table for any
statusvalue outside the source’s permitted vocabulary. An unrecognized token drops the signal from its correlation group and silently shrinks the impact set. - Orphaned signals from tolerance mismatch. Count resolutions returning
None. A spike means the snapping tolerance is tighter than survey accuracy, or one feed drifted onto a different CRS — reconfirm projection alignment before widening the tolerance. - Cross-feeder mis-snaps. Sample resolved features and confirm each signal attached to a feature on its expected feeder. Wrong-feeder matches indicate a tolerance set wider than the spacing between parallel circuits.
- Duplicate events despite the dedup key. Verify the dedup key is computed after resolution, not before. A key built on the raw message rather than the resolved feature cannot collapse re-reports that carry different message ids.
- Stale snapshot. Confirm the resolution layer read a currently validated topology. A snap against a snapshot taken before a field edit resolves signals to features that no longer connect as expected.
Performance & Scale Considerations
Storm-volume telemetry is bursty and heavy-tailed: a quiet feed of a few events per minute becomes tens of thousands per minute when a front moves through, and every one of those must resolve to a feature fast enough to matter. The spatial index is the lever. Keep candidate features in an R-tree or database spatial index and query the index for the small neighborhood around each signal rather than computing distance to every feature — the difference between a resolution that keeps up with the storm and one that falls hours behind. Cache the feature layer in memory and refresh it on the topology-validation cadence, not on every message, so resolution never blocks on a geodatabase read.
Isolate the write path from the system of record. Telemetry lands in a staging table with its own spatial index, and only confirmed events are promoted into the authoritative model, so a burst of provisional signals never holds locks on the production geodatabase during an edit session. Make the consumer idempotent and horizontally scalable: because the dedup key is deterministic, several consumer workers can process partitions of the queue in parallel and an upsert on the key reconciles any overlap. Size the correlation window from the measured skew budget rather than guessing, and log every signal that arrives after its window has closed — a rising late-arrival rate is the early warning that a feed’s latency has degraded. Recomputing impact across the many simultaneous faults a storm produces reuses the same batch machinery described in the parent outage routing and impact automation reference.
Compliance Notes
Telemetry ingestion is the first link in a chain that regulators scrutinize end to end, so its records must be as defensible as the restoration decisions they drive. The reliability metrics utilities publish — SAIDI, SAIFI, and CAIDI — are computed from outage timestamps that originate here, which makes the event-time discipline a compliance requirement and not merely an engineering nicety: the detection time attached to an event must reflect when the physical fault occurred, captured automatically and stored where it cannot be edited after the fact. Every normalized event should therefore persist its source system, origin identifier, event and arrival times, resolved feature, snapping distance, and dedup key as immutable audit metadata.
Because SCADA telemetry crosses the operational-technology boundary into the IT-side GIS, the ingestion path sits squarely inside NERC CIP scope: encrypt telemetry in transit, segment the OT feeds, enforce role-based access to the consumer and staging store, and log access to the system of record. Aligning the pipeline with the NIST Cybersecurity Framework gives a recognized structure for that access control and change management, while authority for the reliability practices themselves should be drawn from primary sources such as the NERC reliability standards. For water systems, the same feature-resolved event discipline underpins the traceable isolation records that AWWA asset-management practice expects behind every pressure-zone shutoff. Treating ingestion as production software — versioned schemas, pinned tolerances, immutable event logs — makes the whole outage record auditable by default.