10 min read2,101 words

Estimating Restoration Times with Historical Outage Data

A restoration estimate is the most visible output an outage system produces. Customers see it, regulators increasingly require it, and dispatchers plan against it — and in many estates it is produced by a single blended average that is wrong for every outage it is applied to. The fix is not a more sophisticated model. It is to decompose by damage class, to use the historical distribution for that class rather than its mean, to add the queue the outage is actually waiting in, and above all to record the basis of every estimate so the next event can improve it. This guide builds that estimator from data an estate already has, and it deliberately publishes a range where the evidence supports only a range. It sits inside the regime discipline set out in storm response and restoration analytics.

Environment Prerequisites

  • Python 3.11 with pandas>=2.0 and numpy>=1.26 in an isolated environment; no geodatabase dependency, since the estimator works on the outage history rather than the network.
  • A restored-outage history carrying, per outage: the damage class, the assignment time, the restoration time, the crew type and source, the season, and the affected count.
  • A damage classification available early. The estimator is only as good as the class it is given, and a class assigned after restoration is useless for prediction.
  • Current crew availability and the queue as the dispatch loop sees them, so the expected start reflects reality rather than an assumption of immediate assignment.
  • A store for published estimates with their basis, timestamped and immutable, because the scoring pass reads it.
  • A minimum sample size per class, below which the estimator must widen to a range instead of publishing a point estimate.

Schema-Aware Validation Protocol — Run Before Publishing Estimates

  1. Confirm the history’s durations measure the same interval. Assignment-to-restoration and report-to-restoration differ by the queue, and mixing them produces a distribution that models neither.
  2. Check the classification is assigned before restoration. A class populated retrospectively makes the model look excellent in backtest and useless in production.
  3. Verify the sample size per class and season. A class with a handful of examples cannot support a point estimate; the estimator must know its own sample size and widen accordingly.
  4. Look for survivorship in the history. Outages that were never restored — cancelled, merged, reclassified — must be excluded explicitly rather than left to skew the distribution.
  5. Confirm the crew-availability input is live. An estimator using a nominal crew count during an event where half the crews are committed will be systematically optimistic, and the error will be blamed on the duration model rather than on the queue.

Minimal Reproducible Implementation

The estimator below builds a duration distribution per damage class and season, combines it with the queue position to produce an expected start, and returns an estimate that carries its own basis. When the sample is thin, it returns a range and says why.

from __future__ import annotations

import logging
from dataclasses import dataclass, field

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("etr")

MIN_SAMPLE = 12          # below this, publish a range rather than a point estimate


@dataclass
class Estimate:
    """A published restoration estimate and everything that produced it."""

    outage_id: str
    damage_class: str
    low_h: float
    high_h: float
    point_h: float | None
    basis: dict = field(default_factory=dict)

    @property
    def is_range(self) -> bool:
        return self.point_h is None


def duration_distribution(history: pd.DataFrame, damage_class: str,
                          season: str) -> np.ndarray:
    """Assignment-to-restoration durations, in hours, for one class and season."""
    sel = history[(history["damage_class"] == damage_class)
                  & (history["season"] == season)
                  & history["restored_at"].notna()]
    hours = (sel["restored_at"] - sel["assigned_at"]).dt.total_seconds() / 3600.0
    return hours.to_numpy()


def estimate_restoration(
    outage_id: str,
    damage_class: str,
    season: str,
    history: pd.DataFrame,
    queue_ahead: int,
    crews_available: int,
    crew_source: str = "HOME",
) -> Estimate:
    """Produce an estimate with its basis attached.

    ``queue_ahead`` is how many comparable jobs sit in front of this one; combined with
    the crews available it converts a duration into an expected start. A thin sample
    yields a range rather than a point estimate, and says so in the basis.
    """
    durations = duration_distribution(history, damage_class, season)
    n = int(durations.size)

    if n == 0:
        LOG.warning("no history for %s/%s — falling back to the class across seasons",
                    damage_class, season)
        durations = duration_distribution(history, damage_class, season="ALL")
        n = int(durations.size)

    if n == 0:
        return Estimate(outage_id, damage_class, low_h=2.0, high_h=24.0, point_h=None,
                        basis={"reason": "no history for this class", "sample": 0})

    p25, p50, p75 = (float(np.percentile(durations, q)) for q in (25, 50, 75))

    # Queue: comparable jobs ahead, divided by the crews that can take them.
    per_wave = max(1, crews_available)
    wait_h = (queue_ahead / per_wave) * p50
    if crew_source != "HOME":
        wait_h *= 1.15          # unfamiliar crews are slower on their first shift

    basis = {
        "sample": n, "p25": p25, "p50": p50, "p75": p75,
        "queue_ahead": queue_ahead, "crews_available": crews_available,
        "crew_source": crew_source, "wait_h": wait_h,
    }

    if n < MIN_SAMPLE:
        basis["reason"] = f"sample below {MIN_SAMPLE} — range published"
        return Estimate(outage_id, damage_class, low_h=wait_h + p25,
                        high_h=wait_h + p75, point_h=None, basis=basis)

    return Estimate(outage_id, damage_class, low_h=wait_h + p25, high_h=wait_h + p75,
                    point_h=wait_h + p50, basis=basis)


def score_estimates(published: pd.DataFrame, actuals: pd.DataFrame) -> pd.DataFrame:
    """Compare each published estimate against the outcome, grouped by damage class."""
    joined = published.merge(actuals, on="outage_id", how="inner")
    joined["error_h"] = (joined["actual_h"] - joined["point_h"])
    return (joined.groupby("damage_class")["error_h"]
            .agg(["count", "mean", "median", "std"])
            .rename(columns={"mean": "bias_h"}))
Median restoration time by damage class, and why one average serves none of them Restoration time separates by damage class far more strongly than by any other variable. A device operation is a switching job measured in minutes once someone reaches it. Single-span damage is a crew-hours job with a predictable spread. Structure damage adds equipment and frequently a second crew. Access-constrained work is governed by something outside the utility — a flooded road, a blocked right of way — and its distribution has a long tail that no average can represent. Publishing one blended estimate is wrong for every class at once. median hours from assignment to restoration, one estate, three seasons Device operation 0.6 h switching, once reached Single-span damage 3.2 h predictable spread Structure damage 7.5 h equipment, second crew Access constrained 19.0 h long tail, external cause Four distributions, not one — and the fourth one has no useful mean.

The score_estimates function is the half that most estates skip, and it is the half that makes the estimator improve. An estimate that is never compared against the outcome is an opinion; one that is scored by class every week is a model with a measured bias that can be corrected.

How a restoration estimate is produced, and what it must carry with it The outage is classified as early as the evidence allows, using the protection operation, the affected count and any field report. Historical durations for that class, that season and that crew type give a distribution rather than a number. The queue position and the crew availability convert it into an expected start, and the estimate published is the start plus the duration. Everything that went into it — class, sample size, queue position, crew source — is stored with the estimate, because an estimate whose basis is not recorded cannot be scored afterwards. CLASSIFY damage class from early evidence DISTRIBUTION historical durations class + season QUEUE position and crew availability PUBLISH start + duration with its basis no evidence yet CLASS UNKNOWN publish a range and say so, rather than a precise number from a blended average An estimate without its basis recorded cannot be improved by the next event.

Production Deployment Pattern

  1. Re-estimate on every material change, not on a timer: a class reassignment, a crew assignment, or a queue change ahead of this outage. Publishing a new number when nothing changed erodes confidence in all of them.
  2. Publish ranges honestly. A range labelled as a range is trusted; a point estimate that turns out to be a guess is not, and the loss of credibility applies to every later estimate.
  3. Never publish an estimate without storing its basis. The basis is what the scoring pass reads, and it is also what answers a complaint about a missed estimate.
  4. Score weekly, in season. Group errors by damage class and crew source, and treat a systematic bias as an input defect. The patterns are stable enough that a season of scoring produces a materially better model.
  5. Feed the classification back. When scoring shows a class with large errors in both directions, that class is holding two populations and should be split — usually by access or by equipment requirement.
  6. Keep the estimator out of the operational path. It reads history and the current queue, and it publishes; it should never be able to block dispatch if it fails.
Scoring published estimates against outcomes, and what each error pattern indicts Systematic optimism in one class means the duration distribution for that class is drawn from the wrong sample — often because easy jobs are classified into it. Systematic pessimism usually means the queue model is too conservative about crew availability. Estimates that are accurate at the start of an event and wrong later indicate the queue model does not account for crews being consumed. And a class whose errors are large in both directions is usually a class that needs splitting, because it contains two populations. Pattern Indicts Fix Optimistic in one class wrong sample for that class audit the classification Pessimistic overall queue model too conservative crew availability inputs Degrades through the event crews consumed, not modelled decay availability Large errors both ways the class holds two populations split the class Every pattern points at an input; none of them points at the crews.

Conclusion

Restoration estimation gets better through decomposition and scoring rather than through sophistication. Splitting by damage class turns one useless average into four usable distributions; adding the queue turns a duration into an arrival time; recording the basis makes the next event an opportunity to improve rather than a repeat. The estimates and their outcomes also become the raw material for the reliability filing that follows the event, which is where post-storm reliability reporting picks up.

For authoritative reference, consult the pandas documentation and the NumPy statistics reference.

Related pages