Incremental Topology Rebuild After Field Edits

A full topology validation of an enterprise utility network is a blunt instrument. When a field crew moves three service points and re-snaps a lateral, the edits touch a few hundred square meters, yet a naive nightly job revalidates the entire feature dataset — tens of thousands of edges — to confirm what those three edits changed. On a large water or electric model the full pass can run for tens of minutes while holding locks that block editors and downstream automation. Worse, running it too rarely lets a backlog of unvalidated edits accumulate, and every trace launched against that backlog reads a stale connectivity graph. The engineering answer is to validate only what changed: the utility network already records each edit as a dirty area, so an incremental rebuild that revalidates just those envelopes gives the same correctness guarantee as a full pass at a fraction of the cost. This is the same batch discipline established across batch topology processing with Python, narrowed to the specific problem of keeping topology validation current after high-frequency field edits.

This page provides a complete, copy-paste arcpy workflow that reads the dirty-area layer, coalesces the flagged envelopes into a small set of processing tiles, runs an extent-scoped ValidateTopology over each tile, exports and counts the residual errors, and gates the network before tracing resumes. Getting this right matters beyond runtime: a partially validated network silently corrupts isolation results, so the barrier states consumed by valve barrier logic depend on the topology being clean everywhere an edit landed — not merely everywhere it was convenient to check.

Environment Prerequisites

An incremental rebuild that skips the wrong extent is more dangerous than no rebuild at all, because it reports success over an area it never touched. Lock the following before running anything:

• ArcGIS Pro 3.2+ with an active Standard or Advanced license; arcpy.un.ValidateTopology and the utility-network dirty-area model require it. Confirm arcpy.CheckExtension reports the utility-network capability as available.
• Python 3.11 from the ArcGIS Pro conda environment (clone it — never mutate the base arcgispro-py3), with arcpy importable from the interpreter that runs the job.
• Enterprise geodatabase connection (.sde) pointing at a named, isolated version — not DEFAULT — so the incremental pass validates committed edits without contending with active edit sessions on the production branch.
• Utility network topology enabled with a populated dirty-area layer. If the topology was disabled and re-enabled, the dirty-area history resets and the first pass after re-enabling must be a full validation before incremental runs are meaningful.
• A defined processing tolerance and tile budget: the network XY tolerance (for example 0.01 m) and a maximum tile envelope size so a single sprawling edit does not degrade the incremental pass back into a near-full validation.
• Write access to a report location — a file geodatabase or a version-controlled directory — for the per-run JSON report that records which envelopes were validated, timings, and residual counts.

Schema-Aware Validation Protocol — Run Before the Rebuild

Most incremental-rebuild failures are not in the validation call; they are in the assumption that the dirty-area set faithfully describes what changed. Work this ordered checklist first — the earliest item is the most frequent culprit.

1. Confirm the dirty-area layer is actually populated. If edits were made through a client that suppresses dirty-area generation, or the topology was validated by another process between your read and your run, the envelope set is empty and the pass validates nothing while returning success. Count the dirty areas first and treat a zero count as a decision point, not a silent no-op.
2. Check for edits that dirty a wider extent than their geometry. Deleting a junction can invalidate connectivity across every edge that referenced it, and the dirty area reflects that broader footprint. Trust the dirty-area envelope over the edited feature’s own extent; validating only the moved geometry misses the connectivity fallout around it.
3. Detect stacked or adjacent envelopes. Dozens of small edits in one work area produce dozens of overlapping envelopes. Validating each independently repeats work and multiplies lock churn. Coalesce touching envelopes into a single tile before validating so each area is revalidated exactly once.
4. Verify the version is reconciled enough to be authoritative. An incremental pass on a child version that is far behind DEFAULT validates geometry that a reconcile will immediately re-dirty. Confirm the version’s lineage is current, or schedule the rebuild to run immediately after reconcile.
5. Bound the worst case. If the union of dirty areas approaches the full dataset extent — after a bulk import or a large re-snap — an incremental pass has no advantage and its per-tile overhead makes it slower. Compare the dirty-area coverage against a threshold and fall back to a single full ValidateTopology when the network is mostly dirty.

Minimal Reproducible Implementation

The following workflow reads the dirty-area layer, coalesces overlapping envelopes into processing tiles, validates each tile with an extent-scoped arcpy.un.ValidateTopology, exports the residual topology errors within each envelope, and returns a structured report. It falls back to a single full validation when the dirty coverage is too large to make the incremental path worthwhile, and it raises when the residual error count breaches the gate.

import arcpy
import json
import logging
from dataclasses import dataclass, asdict, field
from datetime import datetime, timezone
from typing import List

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")


@dataclass
class TileResult:
    """Outcome of validating one coalesced dirty-area tile."""
    xmin: float
    ymin: float
    xmax: float
    ymax: float
    residual_errors: int = 0
    seconds: float = 0.0


@dataclass
class RebuildReport:
    """Structured summary of an incremental rebuild pass."""
    started: str
    mode: str                       # "incremental" or "full-fallback"
    tiles: List[TileResult] = field(default_factory=list)
    total_residual: int = 0
    passed: bool = True


def _collect_dirty_extents(dirty_area_fc: str) -> List[arcpy.Extent]:
    """Read dirty-area envelopes flagged since the last validation."""
    extents: List[arcpy.Extent] = []
    with arcpy.da.SearchCursor(dirty_area_fc, ["SHAPE@"]) as cursor:
        for (shape,) in cursor:
            if shape is not None:
                extents.append(shape.extent)
    logging.info("Collected %d dirty-area envelopes", len(extents))
    return extents


def _coalesce(extents: List[arcpy.Extent], pad: float = 0.0) -> List[arcpy.Extent]:
    """Merge overlapping/touching envelopes so each area is validated once."""
    boxes = [[e.XMin - pad, e.YMin - pad, e.XMax + pad, e.YMax + pad] for e in extents]
    merged: List[list] = []
    for box in sorted(boxes, key=lambda b: b[0]):
        for m in merged:
            overlaps = not (box[0] > m[2] or box[2] < m[0] or box[1] > m[3] or box[3] < m[1])
            if overlaps:
                m[0], m[1] = min(m[0], box[0]), min(m[1], box[1])
                m[2], m[3] = max(m[2], box[2]), max(m[3], box[3])
                break
        else:
            merged.append(list(box))
    return [arcpy.Extent(*m) for m in merged]


def incremental_rebuild(
    utility_network: str,
    dirty_area_fc: str,
    error_out_gdb: str = "in_memory",
    residual_threshold: int = 0,
    full_fallback_ratio: float = 0.6,
) -> RebuildReport:
    """Validate only the dirty envelopes of a utility-network topology.

    Coalesces dirty areas into tiles, runs an extent-scoped ValidateTopology
    over each, counts residual (non-exception) errors, and gates the result.
    Falls back to a single full validation when the network is mostly dirty.
    """
    report = RebuildReport(started=datetime.now(timezone.utc).isoformat(), mode="incremental")
    extents = _collect_dirty_extents(dirty_area_fc)
    if not extents:
        logging.info("No dirty areas; topology already current")
        return report

    full = arcpy.Describe(utility_network).extent
    full_area = max((full.XMax - full.XMin) * (full.YMax - full.YMin), 1e-9)
    dirty_area = sum((e.XMax - e.XMin) * (e.YMax - e.YMin) for e in extents)

    if dirty_area / full_area >= full_fallback_ratio:
        report.mode = "full-fallback"
        logging.warning("Dirty coverage %.0f%% — running one full validation",
                        100 * dirty_area / full_area)
        tiles = [full]
    else:
        tiles = _coalesce(extents)
        logging.info("Coalesced to %d processing tiles", len(tiles))

    for ext in tiles:
        t0 = datetime.now(timezone.utc)
        tile = TileResult(ext.XMin, ext.YMin, ext.XMax, ext.YMax)
        try:
            # Extent-scoped validation: only geometry inside `extent` is rebuilt.
            arcpy.un.ValidateTopology(utility_network, extent=ext)
            err_fc = f"{error_out_gdb}/topo_err_{abs(hash((ext.XMin, ext.YMin))) % 10**8}"
            arcpy.un.ExportTopologyErrors(utility_network, error_out_gdb,
                                          err_fc.split("/")[-1])
            residual = 0
            if arcpy.Exists(err_fc):
                with arcpy.da.SearchCursor(err_fc, ["IsException"]) as cur:
                    residual = sum(1 for (is_exc,) in cur if not is_exc)
            tile.residual_errors = residual
            report.total_residual += residual
        except arcpy.ExecuteError:
            logging.error("Validation failed for tile %s", (ext.XMin, ext.YMin))
            tile.residual_errors = -1
            report.total_residual += 1
        tile.seconds = (datetime.now(timezone.utc) - t0).total_seconds()
        report.tiles.append(tile)

    report.passed = report.total_residual <= residual_threshold
    logging.info("Rebuild %s: %d residual errors across %d tiles",
                 "PASSED" if report.passed else "FAILED",
                 report.total_residual, len(report.tiles))
    if not report.passed:
        raise RuntimeError(
            f"Incremental rebuild gate: {report.total_residual} residual errors "
            f"exceed threshold {residual_threshold}."
        )
    return report


# Example execution
# rep = incremental_rebuild("network.sde/UN", "network.sde/UN_Dirty_Areas")
# print(json.dumps(asdict(rep), indent=2))

The critical design choice is passing extent to ValidateTopology: the tool rebuilds connectivity only within that envelope, so a pass that touches a work area of a few tiles completes in seconds rather than the minutes a full-network validation demands. Counting only IsException == False errors is what makes the gate trustworthy — accepted exceptions (a known dead-end main, a permitted overlap) must not fail an otherwise clean rebuild, while genuine dangles and connectivity breaks do. The fallback ratio protects the pathological case: after a bulk import that dirties most of the dataset, the per-tile overhead of the incremental path is pure waste, so the routine collapses to a single full validation and labels the report accordingly.

Production Deployment Pattern

A notebook that validates dirty areas is a diagnostic; an enforced, reported rebuild is an engineering control. Promote it into the lifecycle as follows:

1. Run against a reconciled, isolated version. Trigger the rebuild immediately after the version reconciles against DEFAULT, so the envelopes you validate reflect the same lineage a subsequent post will publish. Running before reconcile validates geometry a merge will re-dirty, wasting the pass.
2. Wire the gate into CI/CD. Invoke incremental_rebuild from the same batch runner that drives the rest of batch topology processing with Python; let the raised RuntimeError fail the job on an over-threshold residual count, and parse the returned report to open remediation tickets for each offending tile without blocking unrelated merges.
3. Apply bounded retry on transient locks. Enterprise geodatabase validation can fail intermittently when an editor briefly holds a lock in the target envelope. Wrap the per-tile ValidateTopology in three attempts with exponential backoff so a momentary lock produces a retry rather than a false failure.
4. Persist the report as an audit artifact. Serialize the RebuildReport — tiles, timings, residual counts, mode, and the arcpy version — to a timestamped, version-controlled JSON file. This record shows precisely which envelopes were revalidated and when, which is exactly what a reliability or rate-case review needs to confirm the topology was current when a trace was run.
5. Feed the clean result forward. Only after the gate passes should downstream automation resume — isolation traces, impact analysis, and the valve barrier logic that reads barrier states all assume a validated topology, so the rebuild’s pass signal is their release gate.

Conclusion

Incremental rebuild replaces an all-or-nothing nightly validation with a targeted pass that revalidates only the envelopes field edits dirtied, gates on the residual error count, and produces a per-tile audit record. It delivers the same correctness guarantee as a full validation wherever an edit landed, at a runtime proportional to the change rather than the network size, and it degrades gracefully to a full pass when the dataset is mostly dirty. Enforcing it keeps tracing honest between full validations and prevents the stale-graph traces that a validation backlog invites. The natural next step is to schedule the rebuild directly off the edit-commit event so the topology is current within seconds of a crew posting its work.

Related

• Up to the parent topic: Batch Topology Processing with Python
• Up to the section: Topology & Tracing Workflows
• Batch Processing Topology Errors Using ArcPy and GeoPandas
• Valve & Isolator Mapping Strategies

For authoritative reference, consult the ArcGIS Pro Validate Topology (Utility Network) tool documentation and the topology in ArcGIS overview.