10 min read2,263 words

Diagnosing Subnetwork Controller Drift

Controller drift is the slow divergence between the sources a utility network model believes feed each subnetwork and the sources that actually do. It is produced by ordinary work: a tie closed during a storm and left closed, a regulator replaced under a work order that changed the asset but not the controller, a substation reconfiguration modelled as a geometry edit. None of these raise an error, because the model remains perfectly consistent with itself — it stamps membership from the controllers it holds, and it holds the ones it was given. The consequence only surfaces when a report, a switching plan or a crew disagrees with the map, typically weeks later. This guide builds the scheduled comparison that finds drift from the model alone, without a field visit, and turns it into a work queue rather than an annual surprise. It complements the change-time gating in updating subnetworks after controller changes, which prevents new drift but says nothing about what has already accumulated.

Environment Prerequisites

  • ArcGIS Pro 3.2+ with a Standard or Advanced licence and arcpy importable from Python 3.11, so the subnetworks table and the controller records can be read.
  • Read access to a reconciled, unversioned snapshot or a read-only replica. The comparison must run against a stable state; reading a live edit version produces findings that describe someone’s work in progress.
  • A persisted history of previous runs — a small table or a version-controlled file holding, per subnetwork, the feature count and last-updated timestamp from each pass. The comparison is a diff, and a diff needs a previous state.
  • The edit log or version reconcile history for the affected extent, so an update older than the last edit can be identified as stale rather than reported as wrong.
  • A work-order feed or a change register listing controller changes that were authorised, so an expected movement is not reported as drift.
  • A defect sink — a ticket endpoint or a table — so each finding becomes work rather than a line in a log nobody reads.

Schema-Aware Validation Protocol — Run Before the Comparison

  1. Confirm the snapshot is reconciled and quiet. A comparison against a version with active edits reports the editor’s work as drift, which trains everyone to ignore the report.
  2. Confirm every tier in scope has completed at least one update. A tier that has never been walked has no membership to compare, and its first pass will report every subnetwork as changed.
  3. Reconcile the controller list against the feature classes. Every controller record must resolve to a feature that exists and is enabled. This check alone finds the retirements that removed a source without reassigning it.
  4. Normalise subnetwork names before diffing. A rename between passes looks identical to a subnetwork disappearing and another appearing. Where renames are legitimate, carry a mapping so the diff reports a rename rather than a pair of phantom changes.
  5. Load the authorised-change register first. Any movement that matches an approved controller change is expected and should be recorded as such, not raised. A comparison that cannot tell authorised change from drift produces a queue nobody can triage.

Minimal Reproducible Implementation

The routine below reads the current membership and controller state, compares it against the last persisted pass, subtracts anything an authorised change explains, and classifies what remains into the four drift signatures. It returns a structured report and never writes to the network it is auditing.

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from datetime import datetime

import arcpy

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("controller-drift")

DRIFT_UNRECORDED = "UNRECORDED_CHANGE"
DRIFT_MISSING_CONTROLLER = "MISSING_CONTROLLER"
DRIFT_STALE_UPDATE = "STALE_UPDATE"
DRIFT_ORPHAN_STAMP = "ORPHAN_STAMP"


@dataclass
class Finding:
    subnetwork: str
    tier: str
    kind: str
    detail: str


@dataclass
class DriftReport:
    findings: list[Finding] = field(default_factory=list)
    compared: int = 0
    explained: int = 0

    @property
    def ok(self) -> bool:
        return not self.findings


def read_state(un_path: str, tiers: set[str]) -> dict[str, dict]:
    """Current membership, controller and update time per subnetwork in the given tiers."""
    state: dict[str, dict] = {}
    fields = ["SUBNETWORKNAME", "TIERNAME", "FEATURECOUNT", "LASTUPDATESUBNETWORK",
              "SUBNETWORKCONTROLLERNAME", "ISDIRTY"]
    with arcpy.da.SearchCursor(f"{un_path}_Subnetworks", fields) as cursor:
        for name, tier, count, updated, controller, dirty in cursor:
            if tier not in tiers:
                continue
            state[name] = {
                "tier": tier,
                "count": int(count or 0),
                "updated": updated,
                "controller": controller,
                "dirty": bool(dirty),
            }
    return state


def live_controllers(un_path: str) -> set[str]:
    """Controller names that still resolve to an existing, enabled feature."""
    live: set[str] = set()
    with arcpy.da.SearchCursor(f"{un_path}_Controllers",
                               ["SUBNETWORKCONTROLLERNAME", "ISENABLED"]) as cursor:
        for name, enabled in cursor:
            if enabled:
                live.add(name)
    return live


def detect_drift(
    current: dict[str, dict],
    previous: dict[str, dict],
    controllers: set[str],
    last_edit: dict[str, datetime],
    authorised: set[str],
    tolerance: float = 0.02,
) -> DriftReport:
    """Classify the difference between two passes into drift signatures.

    ``authorised`` holds subnetwork names with an approved controller change since the
    previous pass; movement on those is expected. ``last_edit`` maps a tier to the
    timestamp of its most recent posted edit, which is what makes an update stale.
    """
    report = DriftReport()

    for name, now in current.items():
        report.compared += 1
        tier = now["tier"]

        if now["controller"] and now["controller"] not in controllers:
            report.findings.append(Finding(
                name, tier, DRIFT_MISSING_CONTROLLER,
                f"controller {now['controller']} no longer resolves to an enabled feature"))
            continue

        edited = last_edit.get(tier)
        if edited and now["updated"] and now["updated"] < edited:
            report.findings.append(Finding(
                name, tier, DRIFT_STALE_UPDATE,
                f"last updated {now['updated']}, last edit {edited}"))
            continue

        was = previous.get(name)
        if was is None:
            continue  # new subnetwork: nothing to compare against yet

        if name in authorised:
            report.explained += 1
            continue

        base = max(1, was["count"])
        moved = abs(now["count"] - was["count"]) / base
        if moved > tolerance:
            report.findings.append(Finding(
                name, tier, DRIFT_UNRECORDED,
                f"membership {was['count']} -> {now['count']} with no authorised change"))

    for name in previous:
        if name not in current:
            report.findings.append(Finding(
                name, previous[name]["tier"], DRIFT_ORPHAN_STAMP,
                "subnetwork present in the previous pass and absent now"))

    LOG.info("compared %d subnetwork(s): %d finding(s), %d explained",
             report.compared, len(report.findings), report.explained)
    return report
How controller drift accumulates between the change and the discovery A switching operation moves a tie into service and nobody records a controller change, because the physical work needed none. The nightly update stamps membership from the controllers the model still holds, so the model stays internally consistent and externally wrong. Weeks later a reliability report attributes an outage to the wrong circuit, and the investigation traces back to the switching operation. The interval between the two is the window in which every decision made on that membership was built on a stale source. Field switching tie closed, source changed day 0 No model change controllers untouched day 0 Update runs stamps the old membership nightly Report disagrees outage on the wrong circuit week 6 Root cause drift found by hand week 6 every day in this interval produced decisions on a stale source Nothing in the pipeline fails; the model is consistent with itself throughout.

The classification matters more than the detection. A stale update and an unrecorded change look identical in a raw count comparison, and they need entirely different responses: one is a scheduling problem, the other is a field-to-model reconciliation. Separating them is what makes the queue actionable.

Four drift signatures a scheduled comparison can detect without field confirmation A subnetwork whose feature count moves without a recorded controller change points at a field switching operation the model never learned about. A controller whose feature no longer exists means a retirement removed the source and the tier has been walking without it. A subnetwork that has not been updated since before the last edit in its extent is stale rather than wrong. And a feature carrying a subnetwork name for a tier that no longer lists it is a leftover stamp from a tier that was reorganised. Signature What it means Detect by Count moved, no change recorded unrecorded field switching diff counts run to run Controller feature missing source retired out from under the tier join controllers to features Update older than the last edit stale, not wrong timestamps vs edit log Name for a tier that no longer lists it orphan stamp from a reorganised tier names not in the tier list All four are computable from the model alone — no field visit required.

Production Deployment Pattern

  1. Run it nightly against the reconciled snapshot, immediately after the scheduled subnetwork update completes, so a stale-update finding means the schedule failed rather than that the comparison ran too early.
  2. Persist every pass, not only the findings. The state table is the previous pass for tomorrow’s run, and the history is what shows the drift rate falling after a process change.
  3. Feed the authorised-change register from the work-management system. A controller change raised as a work order and applied through the gated routine should arrive in the register automatically; anything that does not is drift by definition.
  4. Route findings by kind. A missing controller is an engineering ticket. An unrecorded change is a field reconciliation. A stale update is an operations ticket against the schedule. Sending all three to one queue guarantees two of them wait behind the wrong specialists.
  5. Alarm on the rate, not the count. The first passes clear a historical backlog and their volume says nothing about current process health. Alarm when the steady-state rate rises.
  6. Persist an audit record per pass — tiers compared, findings by kind, explained count, the snapshot moment, and the tool version — so the drift history is reconstructable during a reliability review.
Drift found per week by a scheduled comparison, by cause When a comparison first runs against an estate that has never had one, most of what it finds is historical: switching operations that were never modelled, controllers removed during retirements, tiers reorganised years ago. Those clear out over the first few passes. What remains is the steady-state rate, and that number is the useful one: it measures how often the field and the model diverge, which is a process measurement rather than a data one. illustrative shape of a first-run backlog clearing First pass — historical 46 findings accumulated backlog Second pass 11 findings remainder of the backlog Steady state 2 findings the real divergence rate The steady-state number is the process metric; the first pass is just the backlog.

Conclusion

Controller drift is not a data-quality problem that a cleanup sprint fixes; it is the visible residue of field work that never reached the model. A nightly comparison that classifies the difference between two passes turns it into a measurable rate with an owner per category, and the rate is what improves when the switching process starts recording controller changes. Run it long enough and the interesting output stops being the findings and becomes the trend. The next step is to make the tier structure itself explicit, because a drift comparison across tiers is only as meaningful as the tier definitions underneath it.

For authoritative reference, consult the ArcGIS Pro subnetwork documentation and the Python datetime library.

Related pages