13 min read2,950 words

Storm Response & Restoration Analytics

Everything an outage automation stack does well in normal operation it does badly in a storm, and the reason is not capacity. It is that each component is tuned against an assumption that stops holding when events outnumber crews: that corroboration is evidence, that a network snapshot stays valid for the duration of an analysis, that an optimal plan is more useful than a stable one. A system that behaves correctly on a Tuesday and produces phantom outages, churning dispatch plans and stale impact sets during a wind event is not a system that failed under load — it is a system operating outside the regime it was designed for, without saying so.

This page treats storm operation as an explicit regime with its own parameters, entered and left deliberately. It sets out what changes and why, how restoration analytics differ from ordinary reporting, and what the storm phases owe each other in records — because the reliability filing that follows a major event is assembled from artefacts that only exist if somebody wrote them down at the time. It builds on the detection and impact machinery in outage routing and impact automation and the dispatch loop in crew dispatch and route optimization.

The three regimes an outage system operates in, and what moves it between them In normal operation the system confirms events on absolute corroboration thresholds and re-solves dispatch whenever anything changes. Elevated operation begins when the arrival rate of events exceeds the rate at which crews can be assigned; thresholds move to relative measures and the re-solve cadence is fixed rather than event-driven. Storm mode begins when the network configuration itself stops being stable, and its defining change is that impact sets are re-derived rather than cached. Recovery is the return path, and it is entered deliberately rather than by the rate falling for a few minutes. NORMAL absolute thresholds ELEVATED fixed re-solve cadence STORM impact sets re-derived RECOVERY backlog worked down events outpace assignment configuration unstable arrival rate falls declared, not inferred Entering storm mode is a decision; leaving it should be one too. Each regime changes thresholds, cadence and what may be cached.

Prerequisite Checklist

Core Model: Regimes, Not Load Levels

The instinct is to treat a storm as more of the same, and to respond with more capacity. That misses what actually breaks. Three assumptions fail simultaneously, and each one needs a different answer.

Corroboration stops discriminating. In normal operation, several last-gasp messages on one feeder is strong evidence of a real outage, because most feeders are quiet. During a storm the base rate changes: signals arrive from everywhere, and an absolute threshold confirms events that are noise. Relative thresholds — a rate significantly above this feeder’s current baseline, rather than a count — restore the discrimination. This is a direct consequence of how AMI last-gasp correlation works: the window and the threshold are measurements of the environment, and the environment has changed.

Snapshots go stale in minutes. Impact analysis pins a network snapshot at the fault timestamp and computes from it. That is sound when the configuration is stable. During restoration, crews are switching continuously, so a snapshot taken twenty minutes ago describes a network that no longer exists, and an impact set computed from it can be wrong in both directions — counting customers already restored, missing customers newly transferred onto a faulted section.

Optimal plans become unusable. The dispatch solver’s objective is efficiency, and during a storm the inputs change faster than a crew can read a plan. A plan that is re-solved on every change is, from the crew’s point of view, no plan at all. Fixing the cadence and warm-starting from the previous solution produces a plan that is slightly worse on paper and dramatically better in practice.

The four parameters that change between normal operation and storm mode Corroboration thresholds move from an absolute count of signals to a measure relative to the current arrival rate, because during a storm every feeder is reporting. The dispatch re-solve moves from event-driven to a fixed cadence with warm starts, because a plan that changes every thirty seconds cannot be executed. Impact sets stop being cached, because the network configuration is moving under them. And the reporting emphasis moves from efficiency to the tail: what is not being done and how long it has waited. Parameter Normal Storm mode Why it changes Corroboration threshold absolute count relative to arrival rate everything is reporting Dispatch re-solve on every change fixed cadence, warm start crews need a stable plan Impact set caching cached per event re-derived configuration is moving Reporting emphasis plan efficiency the waiting tail the tail is what is asked about Four dials, all of which are wrong in a storm if they were tuned for a Tuesday.

Step-by-Step Implementation

  1. Define the regimes and their triggers. Write down what elevated and storm mode mean in measurable terms — arrival rate against assignment rate, switching operations per hour — and who declares each. A regime that is entered by a threshold alone will flap.
  2. Parameterise, do not fork. Every parameter that changes between regimes should be a value in configuration, not a separate code path. A storm-mode branch that is exercised twice a year is a branch that does not work.
  3. Switch corroboration to relative thresholds on entry. Keep the absolute thresholds for the normal regime, and hold both under version control so a post-event review can see which was in force.
  4. Fix the dispatch cadence and warm-start the solver. Publish plans on the cadence, not on change, and pass the previous assignment as the initial solution, following the routing pattern in solving crew routing with OR-Tools.
  5. Re-derive impact sets rather than caching them. Attach the as-of stamp to every set and refuse to publish one whose stamp predates the most recent switching operation in its extent.
  6. Publish estimates with their basis. A restoration estimate with no stated basis cannot be improved afterwards; one that records the model, the crew count assumed and the damage class can be compared against the outcome.
  7. Normalise deliberately. Returning the network to its normal configuration is a checklist item with an owner, not something that happens when people get around to it. Every tie left closed is a controller drift finding waiting to be discovered weeks later.

Diagnostic Protocol

  1. Check which regime was in force. Before diagnosing any storm-period behaviour, confirm which parameters were active. Most “the system got it wrong” reports resolve to normal-mode thresholds running during a storm.
  2. Compare impact-set stamps against switching times. An impact set older than the last switching operation in its extent is stale by definition, and that is the first thing to check when counts disagree with the field.
  3. Look for phantom events in the detection log. A cluster of confirmed events with low corroboration counts, all within a few minutes, is the signature of absolute thresholds in a high-rate environment.
  4. Examine the plan churn rate. Count how many times a work order changed crew during the event. A high number means the re-solve cadence was too fast for the work to be executed.
  5. Audit the normalisation checklist. After the event, compare the current switch positions against the normal configuration. Anything still abnormal is either a deliberate deferral with an owner or an oversight.
  6. Reconcile estimates against outcomes. For each restored outage, compare the published estimate against the actual restoration time. A systematic bias is a model input problem, not a crew performance problem.
A storm from first arrival to the reliability filing, and what each phase owes the next The first hour is detection: signals arrive faster than they can be confirmed individually, and the system either has relative thresholds or it invents outages. The next several hours are assignment, where the constraint is crews rather than information. Restoration runs for as long as it runs, and the only useful measure during it is the tail. Recovery is where the network is returned to its normal configuration — the step most often left half-finished, because the pressure is off. The filing that follows is only as good as the records the earlier phases wrote down. Detection relative thresholds or phantom events hour 0 Assignment crews, not information, is the constraint hour 1 Restoration measure the tail, not the average hours 2–n Normalisation return the configuration, fully after Filing only as good as the records kept weeks each phase owes the next one a record it cannot reconstruct later Normalisation is the phase most often left half-done once the pressure is off.

Performance & Scale Considerations

Detection is the component that saturates first, because signal volume scales with the number of affected meters rather than with the number of events. Bucketing by feeder and applying relative thresholds reduces the work; processing each message individually does not scale.

Impact re-derivation is the expensive consequence of refusing to cache. Bound it by recomputing only the sets whose extent contains a switching operation since the last derivation, which is a small fraction of the total in most events.

Dispatch solve time is bounded by the cadence, not the problem. With a fixed budget and a warm start, the solver returns whatever it has when the budget expires. That is the correct trade in a storm and the wrong one on a quiet day, which is another reason the parameters belong in configuration.

Reporting is read-heavy and should never touch the operational path. Tail reporting, estimate reconciliation and the filing extracts all read a snapshot, following the isolation discipline in version versus snapshot isolation.

Compliance Notes

  • Reliability indices computed across a major event are usually filed with and without the event excluded, and the exclusion has to be defensible. Record the regime transitions with timestamps, because the storm window is the basis of that exclusion.
  • Critical-facility obligations do not relax during a storm. The tail report exists partly so that a facility waiting past its service-level window is visible while something can still be done about it.
  • Estimate publication is increasingly regulated in its own right. Persist every estimate published, its basis, and the time it was superseded, so the sequence a customer saw can be reconstructed.
  • Required audit metadata per event: regime transitions and who declared them, thresholds in force, switching operations with timestamps, impact sets with their as-of stamps, estimates with their bases, notifications with confirmations, and the normalisation checklist at close.

Restoration Analytics: The Four Questions Worth Answering

Storm analytics tends to collapse into a dashboard of counts, which answers nothing anybody asks afterwards. Four questions are worth building for, and each needs a different artefact captured during the event rather than reconstructed later.

How many customers were out, over time? This is the curve everything else is derived from, and it is only trustworthy if impact sets carried as-of stamps. Assembled from cached sets it double-counts the customers who were restored and re-counted under a later event, which is the most common reason a post-event curve does not integrate to the reported customer-minutes.

Where did the time actually go? Split each outage into detection, assignment, travel, and repair. Most estates discover that assignment and travel dominate during a storm while repair dominates on a quiet day, and that finding changes where the next investment goes — more crews versus better staging, for instance.

Which estimates were wrong, and in which direction? A restoration estimate is a prediction, and predictions are only worth making if they are scored. Compare each published estimate against the actual restoration time, grouped by damage class. Systematic optimism on one class is a model input to fix, not a crew to blame.

What did the network look like at each moment? Switching during a storm rewrites the configuration repeatedly, and analyses run afterwards need to know which arrangement was in force when. The switching log with timestamps is the artefact, and it is cheap to write during the event and impossible to reconstruct after it.

Damage Classes and Why Estimates Need Them

A single average restoration time is a number that is wrong for every outage it is applied to. The useful decomposition is by damage class, because the classes differ by an order of magnitude and are usually identifiable early.

Device operation — a recloser that locked out with no physical damage — is restored by a switching operation and takes minutes once a crew or a remote operator reaches it. Single-span damage, a downed conductor or a broken service, is a crew-hours job with a predictable spread. Structure damage, a broken pole or a damaged transformer, adds equipment and often a second crew. Access-constrained damage is the class that breaks estimates: the repair is ordinary but the site is behind a flooded road or a blocked right of way, and the duration is governed by something outside the utility entirely.

Classifying early matters more than classifying precisely. A first estimate that says “structure damage, access unknown” is more useful to a customer and to a dispatcher than a precise number computed from an average that mixes all four. The classification also gives the reconciliation afterwards something to group by, which is what turns a season of events into a model rather than a pile of anecdotes.

Mutual Aid and Borrowed Crews

Large events bring crews from other utilities, and every assumption the dispatch loop makes about a crew stops being safe. A borrowed crew has no history in the skills table, no depot in the travel matrix, and no familiarity with the network it is being sent into. The routing model will happily assign it the same way it assigns a home crew, and the plan will be optimal and wrong.

Three adjustments make mutual aid workable. Register borrowed crews with explicit, conservative skill sets rather than inheriting a default, so an order requiring a qualification nobody verified is never assigned to them. Give them a staging location as their depot rather than a home yard that is four hours away, because the travel matrix is what decides whether they are used at all. And pair them where the work is network-specific: an escort who knows the system converts a crew that would otherwise spend its shift finding assets into one that repairs them.

The analytics consequence is that crew-hours during a mutual-aid event are not comparable to crew-hours in normal operation, and any productivity comparison that ignores the distinction will draw the wrong conclusion about the home crews. Record the crew source on every assignment during the event, and the comparison becomes possible rather than misleading.

The same caution applies to the restoration-time model. Estimates calibrated on home crews will be optimistic for borrowed ones on their first shift and roughly correct by their third, as familiarity with the network improves. Where an event runs long enough for that to matter, hold the crew source as an input to the estimate rather than pretending the difference does not exist.

What Normalisation Owes the Model

The end of a storm is the point at which the network model and the field are furthest apart, and the pressure to move on is highest. Two things have to happen before the event is closed.

Every switching operation performed during the event must be reflected in the model, including the ones performed by field crews without a dispatcher in the loop. Anything missed becomes subnetwork controller drift that is discovered weeks later by a report that disagrees with reality.

Every temporary configuration must either be returned to normal or recorded as a deliberate deferral with an owner and a date. A tie left closed because it is convenient is a decision; a tie left closed because nobody checked is a defect. The two are indistinguishable in the data, which is precisely why the normalisation checklist has to be explicit — it is the artefact that separates them.

In this section