Automating Critical-Facility Outage Notifications
When an outage impact set contains a hospital, a water-treatment plant, or a telecom hub, the utility no longer has a customer-service obligation — it has a statutory one, measured against a clock that starts the moment the outage is confirmed. Life-support customers, dialysis centres, and emergency-response facilities carry mandated notification windows, and missing one is a reportable failure independent of how quickly power or pressure is restored. Doing this by hand does not survive a real event: during a storm the confirmed impact set changes minute by minute, a control-room operator scanning a list for critical accounts will miss one, and there is no defensible record of who was told what and when. Automating the path from impact set to confirmed critical-facility notification closes that gap. This page provides a complete, copy-paste Python implementation that filters the impact analysis and affected-customer tracing output for critical-facility flags, applies jurisdictional SLA windows, dispatches notifications, and records confirmation and audit metadata. It draws the same impact set that feeds computing affected-customer counts with NetworkX, and it operates inside the broader outage routing and impact automation loop where notification and dispatch must read from one shared source of truth.
Environment Prerequisites
Lock the following before wiring notifications against production accounts; a gap here means either a missed statutory notice or a spurious alert to a facility that never lost service.
- Python runtime: Python 3.11 in an isolated environment, created with
conda create -n un-notify python=3.11so message-template and timezone behaviour stay reproducible. - Dependencies:
networkx>=3.0,requests>=2.31, andpandas>=2.0. Therequestslibrary carries the transport to the enterprise messaging or CRM endpoint;networkxsupplies the impact-set graph. - Impact set as input: A committed impact record from the affected-customer trace, where each affected service point carries a
facility_typefrom the coded-value domain and ajurisdictionkey. Notifications must never be derived from a raw geometry query — only from the validated impact set. - Critical-facility register: A current table associating each critical service point with a facility type (
HOSPITAL,WATER_PLANT,TELECOM_HUB,LIFE_SUPPORT), a notification endpoint, and a jurisdiction whose statutory window is documented and versioned. - Time discipline: A single clock source in UTC with explicit timezone-aware timestamps, so SLA deadlines are computed unambiguously across daylight-saving transitions and multi-jurisdiction service areas.
- Idempotency key: A stable notification key per facility per outage event, so a re-fired impact evaluation does not re-notify a facility already confirmed.
Schema-Aware Validation Protocol — Run Before Dispatch
Most notification failures are data failures, not transport failures. Work this ordered checklist first; the earliest item is the most frequent and most costly.
- Confirm the facility register is current. A hospital coded
NONE, or a decommissioned plant still flagged, is the highest-cost defect here — the first misses a statutory notice, the second cries wolf. Reconcile the register against the licensing authority’s facility list before an event, never during one. - Verify jurisdiction-to-window resolution. Every critical facility must resolve to a jurisdiction with a defined SLA window. An unmapped jurisdiction leaves the deadline undefined, which the code must treat as a hard failure rather than a default.
- Check endpoint validity. Confirm each facility’s notification endpoint is reachable and its contact record is current. A stale phone or API endpoint produces a dispatch that reports sent but never confirms.
- Guard idempotency. Ensure the notification key is stable across re-evaluations of the same outage. Without it, a churning impact set re-notifies the same facility repeatedly, burying the real changes.
- Confirm the impact set is the source. Notifications must filter the committed impact set, not a live spatial query. A facility that a fragmented topology omitted from the impact set will also be omitted here — which is why upstream topology validation is a precondition of correct notification.
Minimal Reproducible Implementation
The following is a complete, runnable workflow. It filters the impact set for critical facilities, resolves each to its jurisdictional SLA window, dispatches a notification with a bounded retry, and records a per-facility confirmation with the deadline and audit metadata. Undefined jurisdictions and transport failures raise or are logged rather than silently dropping a statutory notice.
import logging
import time
from dataclasses import dataclass, field, asdict
from datetime import datetime, timedelta, timezone
import requests
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
# Statutory notification windows in minutes, keyed by jurisdiction and facility class.
SLA_WINDOWS_MIN = {
("STATE_A", "HOSPITAL"): 30,
("STATE_A", "WATER_PLANT"): 60,
("STATE_A", "TELECOM_HUB"): 120,
("STATE_A", "LIFE_SUPPORT"): 30,
("STATE_B", "HOSPITAL"): 15,
("STATE_B", "LIFE_SUPPORT"): 15,
}
CRITICAL_TYPES = {"HOSPITAL", "WATER_PLANT", "TELECOM_HUB", "LIFE_SUPPORT"}
@dataclass
class NotificationRecord:
"""Auditable per-facility notification outcome."""
service_point: str
facility_type: str
jurisdiction: str
deadline: str
dispatched_at: str | None = None
confirmed_at: str | None = None
status: str = "PENDING"
detail: str = ""
audit: dict = field(default_factory=dict)
def resolve_deadline(jurisdiction: str, facility_type: str,
confirmed_at: datetime) -> datetime:
"""Return the statutory notification deadline, or raise if undefined."""
window = SLA_WINDOWS_MIN.get((jurisdiction, facility_type))
if window is None:
raise KeyError(
f"no SLA window for {jurisdiction}/{facility_type}; refuse to guess"
)
return confirmed_at + timedelta(minutes=window)
def dispatch_notification(endpoint: str, payload: dict,
retries: int = 3, backoff_s: float = 1.5) -> dict:
"""POST a notification with bounded exponential backoff.
Returns the transport response dict; raises requests.RequestException if
every attempt fails so the caller can escalate rather than assume success.
"""
last_exc: Exception | None = None
for attempt in range(1, retries + 1):
try:
resp = requests.post(endpoint, json=payload, timeout=10)
resp.raise_for_status()
return resp.json()
except requests.RequestException as exc: # transient transport failure
last_exc = exc
logging.warning("dispatch attempt %d/%d failed: %s", attempt, retries, exc)
time.sleep(backoff_s * attempt)
raise requests.RequestException(f"notification dispatch failed: {last_exc}")
def notify_critical_facilities(impact_set: list[dict],
event_id: str,
confirmed_at: datetime) -> list[dict]:
"""Notify every critical facility in the impact set within its SLA window.
`impact_set` items carry service_point, facility_type, jurisdiction, and
endpoint. Returns a list of auditable NotificationRecord dicts.
"""
records: list[NotificationRecord] = []
for facility in impact_set:
if facility.get("facility_type") not in CRITICAL_TYPES:
continue
sp = facility["service_point"]
try:
deadline = resolve_deadline(
facility["jurisdiction"], facility["facility_type"], confirmed_at
)
except KeyError as exc:
logging.error("SLA resolution failed for %s: %s", sp, exc)
records.append(NotificationRecord(
service_point=sp,
facility_type=facility["facility_type"],
jurisdiction=facility.get("jurisdiction", "UNKNOWN"),
deadline="UNDEFINED",
status="ESCALATE",
detail=str(exc),
))
continue
rec = NotificationRecord(
service_point=sp,
facility_type=facility["facility_type"],
jurisdiction=facility["jurisdiction"],
deadline=deadline.isoformat(),
)
payload = {
"event_id": event_id,
"service_point": sp,
"facility_type": facility["facility_type"],
"message": f"Confirmed outage affecting {sp}; restoration in progress.",
"idempotency_key": f"{event_id}:{sp}",
}
try:
response = dispatch_notification(facility["endpoint"], payload)
now = datetime.now(timezone.utc)
rec.dispatched_at = now.isoformat()
rec.confirmed_at = response.get("confirmed_at", now.isoformat())
rec.status = "CONFIRMED" if now <= deadline else "LATE"
rec.detail = response.get("channel", "unknown")
except requests.RequestException as exc:
rec.status = "ESCALATE"
rec.detail = str(exc)
logging.error("dispatch to %s escalated: %s", sp, exc)
rec.audit = {
"event_id": event_id,
"confirmed_at": confirmed_at.isoformat(),
"computed_at": datetime.now(timezone.utc).isoformat(),
}
records.append(rec)
on_time = sum(1 for r in records if r.status == "CONFIRMED")
logging.info("critical-facility notifications: %d confirmed on time, %d total",
on_time, len(records))
return [asdict(r) for r in records]
if __name__ == "__main__":
demo_impact = [
{"service_point": "SP-HOSP-01", "facility_type": "HOSPITAL",
"jurisdiction": "STATE_A", "endpoint": "https://example.invalid/notify"},
{"service_point": "SP-RES-88", "facility_type": "NONE",
"jurisdiction": "STATE_A", "endpoint": "https://example.invalid/notify"},
]
out = notify_critical_facilities(
demo_impact, event_id="EVT-2026-07-13-004",
confirmed_at=datetime.now(timezone.utc),
)
for record in out:
print(record["service_point"], record["status"], record["deadline"])The ordinary residential service point SP-RES-88 is skipped because its facility_type is NONE; only the hospital enters the notification path. The example endpoint is intentionally unreachable, so the run exercises the escalation branch: when dispatch_notification exhausts its retries it raises, the record is marked ESCALATE rather than silently lost, and a human is handed the facility with its context. That behaviour — never dropping a statutory notice on a transport error — is the whole point of separating dispatch failure from data failure.
Three design choices make the output defensible under audit. The SLA window is resolved from an explicit jurisdiction-plus-class table and refuses to guess when a mapping is absent, so a missing window becomes a loud escalation rather than a silent default. Every record carries its deadline, dispatch time, and confirmation time, so on-time compliance is provable per facility rather than asserted in aggregate. And the idempotency key ties each notification to the outage event and the service point, so a churning impact set re-fires without re-notifying a facility already confirmed.
The separation of dispatched_at from confirmed_at is more than bookkeeping. A statutory window is satisfied by confirmed receipt, not by a message leaving the utility’s systems, so the two timestamps must be distinct fields: the dispatch time proves the utility acted promptly, while the confirmation time — returned by the messaging platform or the facility’s acknowledgement — is the value compared against the deadline. Collapsing them into one field is a common shortcut that quietly reclassifies a late-confirmed notice as on-time, which is exactly the misstatement an audit is designed to catch. Where a channel cannot return a confirmation at all, the record should carry the dispatch time and a status of LATE or ESCALATE rather than a fabricated confirmation, so the gap is visible rather than papered over. The status field therefore encodes four distinct outcomes — confirmed on time, confirmed late, pending, and escalated — and each is a different obligation for the operator reviewing the event. The following diagram shows the path from impact set to an immutable confirmation record, including the escalation branch.
Production Deployment Pattern
A script that sends messages is a prototype; a compliant service is a governed control. Promote the workflow into the outage pipeline as follows.
- Trigger on the confirmed lifecycle transition. Fire notifications only when the outage moves to the confirmed state, using the same shared impact set that dispatch reads, so a provisional single-signal event does not notify a hospital before the outage is real.
- Run behind an idempotent, durable consumer. Drive the notifier from a queue keyed on the outage event id, so a re-evaluated impact set re-checks facilities without re-notifying those already confirmed, and a transient crash resumes rather than double-sends.
- Enforce the deadline with a watchdog. Schedule a check that marks any facility not confirmed before its deadline as a breach, escalates it to a human, and records the miss — the SLA window is only meaningful if a breach is detected automatically.
- Persist confirmations immutably. Append each
NotificationRecordto an append-only store with its deadline, dispatch, and confirmation timestamps, then forward it to the asset-management system or SIEM that holds the compliance trail, so on-time delivery is provable after the event. - Gate the register and windows in CI/CD. Version the facility register and the SLA-window table, and validate on every change that every critical facility resolves to a defined window and a reachable endpoint, promoting only when the checks pass.
Conclusion
Automating critical-facility notification turns a manual, error-prone scan into a deterministic, auditable control that acts on the same impact set the rest of the outage loop trusts. Filtering for critical types, resolving each facility to an explicit jurisdictional SLA window, dispatching with bounded retry, and recording per-facility confirmations makes statutory compliance provable rather than asserted. Refusing to guess an undefined window and routing every transport failure to human escalation ensures no notice is silently dropped, which is the failure a regulator penalises hardest. The natural next step is to wire the deadline watchdog and the immutable confirmation store so that on-time delivery is enforced and evidenced without an operator in the loop.