10 min read2,093 words

Implementing Fallback Routing When Primary Topology Fails in Utility Network GIS

When a primary electrical, gas, or water network topology suffers a hard fault — a cascading breaker trip, a mainline rupture, or corrupted geometric connectivity — the operational priority flips from optimal load distribution to resilient service continuity, and it flips in seconds. Manual rerouting at that moment does not scale: a dispatcher hand-tracing alternate paths through a fractured graph is slow, non-reproducible, and prone to opening a tie that induces reverse flow or a phase imbalance. For utility engineers and GIS technicians working within Core Utility GIS Fundamentals & Network Models, the built-in subnetwork tracing that normally handles topology validation assumes intact junction-edge associations; the moment those associations break it returns no answer at all. A deterministic fallback routing layer that operates independently of primary connectivity rules is therefore not a convenience but a safety and reliability requirement — it must produce the same auditable switching sequence every time, with no heuristic guesswork during an active outage.

Environment Prerequisites

Pin these before running any of the code below. A mismatched interpreter or an unlicensed extension fails silently or, worse, succeeds against a stale topology cache.

  • ArcGIS Pro 3.1+ with the bundled arcpy interpreter (C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3). The Utility Network solver APIs used here require an Advanced licence and a Utility Network user type.
  • NetworkX 3.0+ (pip install "networkx>=3.0" into a clone of the ArcGIS conda env) for the fallback graph traversal; do not mix it into the base env.
  • An enterprise geodatabase branch-versioned and validated, with the network topology enabled (Disable Network Topology must NOT be active) so association and junction system tables are populated.
  • Read access to the UN_Association table, the junction object class, and the subnetworks table; production runs should use a service account with SELECT only.
  • A defined fallback feature class of normally-open switches, emergency tie-lines, and manual bypass valves, with populated OperationalStatus, RatedCapacity, MaintenanceLock, and RegulatoryApproval attributes.
  • Git or a GeoPackage store for the version-controlled adjacency matrix, plus a CI/CD runner (GitHub Actions, Azure DevOps, or Jenkins) that can rebuild it on work-order changes.

Schema-Aware Failure Isolation Before You Route

Topology failures in enterprise geospatial engines rarely manifest as clean breaks. They surface as trace termination errors, orphaned terminals, or isConnected flag mismatches across subnetwork controllers. Run this ordered diagnostic before invoking any routing logic — the most common cause (a trace that simply terminated early) comes first, so you confirm the primary graph is genuinely fractured rather than misconfigured. This protocol is the rapid-isolation counterpart to the broader failure-mode catalogue in Fallback Routing Logic in Legacy Systems.

Schema-aware failure-isolation decision flow A primary subnetwork trace passes through four ordered diagnostics: trace-termination audit, association-integrity scan, IsConnected flag validation, and subnetwork controller state. If all four pass, routing continues on the primary graph. If any diagnostic detects trace termination, an association mismatch, IsConnected equal to zero, or controller drift, the flow branches to invoke the fallback adjacency matrix. Primary subnetwork trace runs 1 · Trace-termination audit arcpy.GetMessages() · barrier vs. fracture trace terminated pass 2 · Association-integrity scan UN_Association · From/ToTerminal check mismatch pass 3 · Connectivity flag validation IsConnected = 0 + Lifecycle = Active IsConnected = 0 pass 4 · Subnetwork controller state SubnetworkName ↔ controller alignment controller drift all pass Continue on primary graph normal impedance-optimised routing Invoke fallback adjacency matrix
  1. Trace termination audit. Run arcpy.un.Trace and inspect arcpy.GetMessages() for non-zero termination indicators or barrier-encounter warnings. A trace that returns early on a barrier is a configuration issue, not a fracture — rule it out first.
  2. Association integrity scan. Query the UN_Association table for records where AssociationType is 0 (structural) or 1 (connectivity) and cross-check FromTerminal/ToTerminal values for mismatches against the device terminal configuration.
  3. Connectivity flag validation. Filter junction records where IsConnected = 0 but LifecycleStatus = 'Active'. These orphaned nodes are the structural signature of a fractured graph and must be isolated before routing. The same orphaned-junction pattern is handled in bulk by batch topology error processing.
  4. Subnetwork controller state. Verify SubnetworkName and controller-device alignment. Divergence indicates controller drift or a stale topology cache — re-run subnetwork update before trusting any trace result.

Cross-reference every anomaly against asset lifecycle status so routing never attempts a path through decommissioned or maintenance-locked infrastructure. When these conditions trigger, the engine bypasses the primary graph and invokes the fallback adjacency matrix; log each diagnostic state to a central telemetry stream for post-incident forensics and continuous topology health scoring.

Dual-graph rationale

Unlike SCADA-driven pathfinding, the fallback layer relies on two graphs. The primary graph handles normal operations, impedance optimization, and real-time telemetry. The secondary graph is a static, pre-computed adjacency list of emergency tie-lines, normally-open switches, and manual bypass valves — version-controlled, synchronized to work orders, and strictly filtered to exclude compliance-locked assets. Decoupling emergency routing from live topology validation is what guarantees sub-second path computation even when the primary model is partially corrupted or mid-batch-update. Build the secondary graph in a lightweight, schema-agnostic form that maps directly to GIS feature datasets:

  1. Extract emergency tie-lines. Query the edge feature class for OperationalStatus = 'Normally Open' and Criticality >= 2; export GlobalID, FromTerminal, ToTerminal, and RatedCapacity.
  2. Build the adjacency list. Map each tie-line to a directed edge and assign static impedance from conductor size, pipe diameter, or valve type.
  3. Apply compliance filters. Exclude edges tagged MaintenanceLock = 1, DecommissionDate < CURRENT_DATE, or RegulatoryApproval = 0.
  4. Version-control and sync. Store the matrix as Git-tracked JSON or GeoPackage and trigger automated rebuilds via CI/CD when work orders change switch states.

Minimal Reproducible Implementation

The following is copy-paste-ready. It builds a fallback-aware graph from GIS feature classes, injects normally-open tie-lines, and computes a capacity-constrained alternate path, returning structured output and handling the failure modes you will actually hit during an incident. The traversal uses NetworkX; the feature-class reads use arcpy.

import networkx as nx
import arcpy
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")


def build_fallback_graph(primary_edges_fc: str, fallback_switches_fc: str) -> nx.MultiDiGraph:
    """Construct a dual-graph fallback routing matrix from GIS feature classes."""
    G = nx.MultiDiGraph()

    # Load primary topology edges for reference and impedance baseline
    with arcpy.da.SearchCursor(
        primary_edges_fc, ["OID@", "GlobalID", "Shape_Length", "MaterialType"]
    ) as cursor:
        for oid, gid, length, material in cursor:
            G.add_node(f"edge_{oid}", globalid=gid, length=length, material=material)

    # Inject emergency tie-lines and normally-open switches as traversable edges
    with arcpy.da.SearchCursor(
        fallback_switches_fc,
        ["OID@", "FromTerminal", "ToTerminal", "RatedCapacity", "OperationalStatus"],
    ) as cursor:
        for oid, from_t, to_t, capacity, status in cursor:
            if status == "Normally Open" and capacity and capacity > 0:
                G.add_edge(
                    from_t,
                    to_t,
                    switch_oid=oid,
                    capacity=capacity,
                    weight=1.0 / capacity,  # Inverse capacity: prefer high-throughput paths
                    operational=True,
                )

    logging.info(
        "Fallback graph initialized: %d nodes, %d edges",
        G.number_of_nodes(),
        G.number_of_edges(),
    )
    return G


def compute_fallback_route(
    G: nx.MultiDiGraph, source_terminal, target_terminal, max_capacity_threshold: float
) -> dict:
    """Compute a constrained alternate path using capacity-aware Dijkstra."""
    try:
        # Restrict traversal to edges that meet the capacity threshold
        valid_edge_keys = [
            (u, v, k)
            for u, v, k, d in G.edges(data=True, keys=True)
            if d.get("capacity", 0) >= max_capacity_threshold
        ]
        subgraph = G.edge_subgraph(valid_edge_keys).copy()

        path = nx.shortest_path(
            subgraph, source=source_terminal, target=target_terminal, weight="weight"
        )
        path_edges = list(zip(path[:-1], path[1:]))

        # Extract operational metadata for each edge in the path
        route_metadata = []
        for u, v in path_edges:
            edge_data = G.get_edge_data(u, v) or {}
            for _key, data in edge_data.items():
                if data.get("operational"):
                    route_metadata.append(
                        {
                            "switch_oid": data["switch_oid"],
                            "capacity": data["capacity"],
                            "impedance": data["weight"],
                        }
                    )

        return {"path": path, "metadata": route_metadata, "status": "SUCCESS"}

    except nx.NetworkXNoPath:
        logging.warning("No viable fallback path exists within capacity constraints.")
        return {"path": None, "metadata": [], "status": "NO_PATH"}
    except nx.NodeNotFound as exc:
        logging.error("Terminal node not found in fallback graph: %s", exc)
        return {"path": None, "metadata": [], "status": "NODE_NOT_FOUND"}
    except Exception as exc:  # final guard so an outage script never dies silently
        logging.error("Fallback routing failed: %s", exc)
        return {"path": None, "metadata": [], "status": "ERROR"}

A few implementation notes worth carrying into review. The weight uses inverse capacity (1.0 / capacity) so the solver prefers high-throughput tie-lines, which aligns with utility load-balancing practice. Edge filtering happens before traversal so the constrained search never wastes cycles on under-rated edges during incident response. The three distinct return statuses — NO_PATH, NODE_NOT_FOUND, and ERROR — let the calling orchestration distinguish “the network genuinely cannot reroute” from “the terminal id was wrong” from “the database threw.” For multi-objective routing or k-shortest alternates, see the official NetworkX shortest-path documentation.

Production Deployment Pattern

Run the routing engine as a stateless microservice that consumes a topology snapshot, executes the constrained traversal, and publishes a switching sequence to SCADA or OMS — never as an ad-hoc script on an analyst’s machine. Wrap the traversal in a retry loop with exponential backoff to ride out transient database locks during high-concurrency outage events, and apply the following validation and rollback protocol on every activation:

  1. Pre-execution simulation. Run the candidate path against a read-only topology snapshot. Confirm every required switch exists, is reachable, and has a valid remote-control endpoint.
  2. Telemetry handshake. Before issuing open/close commands, poll real-time voltage, pressure, or flow sensors at the target subnetwork and confirm the path will not induce reverse flow, phase imbalance, or a pressure surge.
  3. Ordered switching sequence. Dispatch commands in topological order (source → intermediate → target) with per-command acknowledgment timeouts; auto-rollback if any switch fails to report a state change inside the SLA window.
  4. Post-incident reconciliation. Once the primary topology is restored, run the reverse switching sequence and log every fallback activation to the asset lifecycle database for reliability scoring and predictive maintenance.

Feed the same Git/CI/CD pipeline that rebuilds the adjacency matrix: when a CMMS work order changes a switch state, trigger a matrix rebuild, re-run the simulation gate, and fail the build on any unreachable required switch. For trace and barrier configuration during the recovery phase, consult the official ArcGIS Pro utility network trace configuration documentation.

Conclusion

This procedure automates the most time-critical moment in network operations: detecting a fractured primary graph through schema-aware diagnostics, pivoting to a version-controlled fallback adjacency matrix, and computing a capacity-constrained alternate path that publishes an auditable switching sequence. Doing it programmatically turns network resilience from a reactive, error-prone manual task into a repeatable automation layer that satisfies reliability-reporting and audit obligations because every reroute is logged and reproducible. The natural next step is to wire these triggers into your outage workflows so topology-degradation metrics invoke the fallback layer before a dispatcher ever has to. Continue with batch topology processing with Python to keep the underlying graph clean between incidents.