10 min read2,145 words

Post-Storm Reliability Reporting with Python

The reliability filing that follows a major event is assembled weeks later from records that cannot be recreated. Nobody can go back and observe how many customers were out at 03:40, or which switching arrangement was in force when a feeder was re-energised, or what estimate a customer was given before it was revised. Either those artefacts were written down during the event or the filing is reconstructed from the final state of the outage table, which is a different and smaller thing. This guide sets out how to build the report from the event log, how to integrate a defensible customer-minutes curve from stamped impact sets, how to apply a major-event exclusion by method rather than by judgement, and how to seal the result so it can be defended. It is the last phase of the sequence described in storm response and restoration analytics.

Environment Prerequisites

  • Python 3.11 with pandas>=2.0 and numpy>=1.26. The reporting layer reads records, not the network, so it needs no geodatabase licence.
  • An immutable event log covering the reporting period: outage confirmations, impact sets with as-of stamps, switching operations with timestamps, estimates with their bases, and notifications with confirmations.
  • The served-customer base for each reporting unit, taken from the customer information system as at the reporting period, not as at today.
  • The published major-event methodology your jurisdiction requires, implemented rather than approximated, so the exclusion is reproducible by a reviewer.
  • A write-once store for the sealed report, its hashes, and the parameter set used.
  • A reconciled snapshot of the network for any spatial roll-up, following the isolation pattern in version versus snapshot isolation.

Schema-Aware Validation Protocol — Run Before Computing Anything

  1. Confirm every impact set carries an as-of stamp. Sets without one cannot be placed on the timeline, and integrating them produces a curve that double-counts restored customers. This is the single most common defect in a reconstructed filing.
  2. Check the served-customer base is period-correct. Using today’s base against last quarter’s interruptions changes every index by the growth in between.
  3. Reconcile outage start and end against the switching log. An outage whose end predates the switching operation that restored it indicates a clock or a data-entry problem that will otherwise distort the curve.
  4. Verify the major-event method’s inputs. The threshold is computed from a historical distribution of daily index values; confirm the history used matches the method’s definition, including how it treats prior major events.
  5. Confirm the reporting boundary. Interruptions that span midnight, or the start or end of the period, must be apportioned by an explicit rule rather than by whichever side the record happens to fall on.

Minimal Reproducible Implementation

The routine below integrates the impact curve from stamped sets, computes the standard indices, applies a major-event exclusion by the published method, and returns a sealed record.

from __future__ import annotations

import hashlib
import json
import logging
from dataclasses import dataclass, field

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("reliability")


@dataclass
class Indices:
    saidi: float
    saifi: float
    caidi: float
    customers_served: int
    customer_minutes: float
    customers_interrupted: int
    excluded_days: list[str] = field(default_factory=list)


def customer_minutes(events: pd.DataFrame) -> pd.Series:
    """Customer-minutes per day, integrated from stamped impact sets.

    ``events`` carries one row per impact set: outage id, as_of, and affected count.
    The count is treated as constant from its stamp until the next stamp for the same
    outage, which is what makes a partial restoration reduce the curve at the moment it
    happened rather than at the end.
    """
    if events["as_of"].isna().any():
        raise ValueError("impact sets without an as_of stamp cannot be integrated")

    ordered = events.sort_values(["outage_id", "as_of"]).copy()
    ordered["next_as_of"] = (ordered.groupby("outage_id")["as_of"].shift(-1)
                             .fillna(ordered["restored_at"]))
    ordered["minutes"] = ((ordered["next_as_of"] - ordered["as_of"])
                          .dt.total_seconds() / 60.0)
    ordered["cmi"] = ordered["minutes"] * ordered["affected"]
    ordered["day"] = ordered["as_of"].dt.date.astype(str)
    return ordered.groupby("day")["cmi"].sum()


def major_event_threshold(daily_saidi: pd.Series) -> float:
    """Threshold from the published method: the log-normal 2.5-beta of daily values."""
    positive = daily_saidi[daily_saidi > 0]
    if positive.size < 30:
        LOG.warning("only %d day(s) of history — threshold is weakly supported",
                    positive.size)
    logs = np.log(positive.to_numpy())
    return float(np.exp(logs.mean() + 2.5 * logs.std(ddof=1)))


def compute_indices(events: pd.DataFrame, customers_served: int,
                    daily_history: pd.Series, exclude_major: bool = True) -> Indices:
    """Compute SAIDI, SAIFI and CAIDI, optionally excluding major-event days."""
    per_day = customer_minutes(events)
    excluded: list[str] = []

    if exclude_major:
        threshold = major_event_threshold(daily_history)
        daily_saidi = per_day / customers_served
        excluded = sorted(daily_saidi[daily_saidi > threshold].index.tolist())
        per_day = per_day.drop(index=excluded, errors="ignore")
        LOG.info("major-event threshold %.2f excluded %d day(s)", threshold, len(excluded))

    kept = events[~events["as_of"].dt.date.astype(str).isin(excluded)]
    interrupted = int(kept.groupby("outage_id")["affected"].max().sum())
    cmi = float(per_day.sum())

    saidi = cmi / customers_served
    saifi = interrupted / customers_served
    caidi = (saidi / saifi) if saifi else 0.0

    return Indices(saidi=saidi, saifi=saifi, caidi=caidi,
                   customers_served=customers_served, customer_minutes=cmi,
                   customers_interrupted=interrupted, excluded_days=excluded)


def seal(indices: Indices, method: str, period: str, source_moment: str) -> dict:
    """Produce the immutable record a reviewer can verify."""
    payload = {
        "period": period, "method": method, "source_moment": source_moment,
        "saidi": round(indices.saidi, 4), "saifi": round(indices.saifi, 4),
        "caidi": round(indices.caidi, 4),
        "customer_minutes": round(indices.customer_minutes, 1),
        "customers_interrupted": indices.customers_interrupted,
        "customers_served": indices.customers_served,
        "excluded_days": indices.excluded_days,
    }
    blob = json.dumps(payload, sort_keys=True).encode("utf-8")
    payload["sha256"] = hashlib.sha256(blob).hexdigest()
    return payload
Assembling a reliability filing from the artefacts an event produced The filing starts from the event log rather than from the outage table, because the log records what was true at each moment while the table records the final state. Customer minutes are integrated from impact sets with as-of stamps, so restorations are subtracted at the moment they happened. Major-event days are identified by the published method rather than by judgement, and the indices are computed twice — with and without them. Every figure carries the version moment and the parameter set that produced it. EVENT LOG what was true at each moment INTEGRATE customer minutes from stamped sets CLASSIFY DAYS major-event method not judgement COMPUTE TWICE with and without major events SEAL moment, method, parameters the classic error CACHED SETS an impact set without an as-of stamp double-counts customers restored and re-counted later The final outage table cannot reproduce the curve; only the stamped log can.

The guard at the top of customer_minutes is the important line. An impact set without an as-of stamp cannot be placed on the timeline, and integrating it anyway produces a curve that looks plausible and overstates the duration index — which is precisely the error a reviewer is most likely to find.

The reporting window and the artefacts that must already exist when it opens By the time a filing is assembled, the event is weeks past and nothing new can be observed. Every input has to have been captured while it was happening: the impact curve from stamped sets, the switching log with timestamps, the estimates with their bases, and the notifications with their confirmations. The filing window is therefore an assembly exercise, and the only way it goes badly is if one of those artefacts was never written. Stamped impact sets the curve is built from these during Switching log configuration at each moment during Estimates + bases what customers were told during Assembly no new observation possible week 2 Filing sealed with method and moment week 4 nothing can be observed after the fact — only assembled The filing is only as good as the artefacts the event was disciplined enough to write.

Production Deployment Pattern

  1. Run the assembly against the write-once event store, never against the live outage system, so a later correction to an outage record cannot silently change a filed figure.
  2. Compute both index sets every time. With and without major-event exclusion, in the same run, so the effect of the exclusion is visible rather than inferred.
  3. Seal every run. The hash, the method name, the period and the source moment turn a spreadsheet into an artefact a reviewer can verify against the inputs.
  4. Re-run historical periods when the method changes. A methodology change that is applied only prospectively makes the series discontinuous; recomputing the history under the new method and publishing both is what keeps it comparable.
  5. Publish the estimate-accuracy figure internally. It is rarely required and it is the number that improves the next event, because it measures communication rather than restoration.
  6. Keep the reporting job entirely read-only. It should be impossible for a reporting run to modify an outage record, which is both an audit requirement and a way to make re-running it safe.
The four figures a post-event review is built around Customer-minutes interrupted is the integral of the impact curve and the basis of the duration index. Customers interrupted is the count that feeds the frequency index. The major-event threshold is computed from the published method rather than chosen, and it decides which days are excluded. The estimate accuracy figure is the one that is optional in most jurisdictions and the most useful internally, because it is the only number that measures how well the utility communicated during the event. CMI customer-minutes integral of the curve CI customers interrupted frequency index input TMED major-event threshold computed, not chosen ± estimate accuracy how well we communicated Three are required; the fourth is the one that improves the next event. All four are assembled from artefacts, not recomputed from the current state.

Conclusion

A defensible reliability filing is an assembly job whose inputs were captured during the event. Integrating the customer-minutes curve from stamped impact sets, applying the major-event exclusion by the published method rather than by judgement, computing both index sets and sealing the result with its hashes turns the filing from an annual argument into a reproducible artefact. The discipline that makes it possible is upstream: stamping impact sets, logging switching operations and recording estimate bases while the event is in progress. Those are the same artefacts the storm response regime asks for, and this is the payoff for having them.

For authoritative reference, consult the IEEE 1366 reliability indices standard and the pandas documentation.

Related pages