8 min read1,858 words

Batch Processing Topology Errors Using arcpy and geopandas

Utility Network GIS deployments accumulate spatial inconsistencies during field data collection, CAD-to-GIS conversions, and legacy migrations, and when rules such as must not overlap, must be covered by, or must not self-intersect are violated across hundreds of thousands of features, fixing them one error at a time in an editing session is neither auditable nor repeatable — a single missed defect can break a downstream trace or an isolation plan. Manually triaging an error inspector window does not scale, cannot distinguish a genuine defect from a tolerance artifact, and leaves no record of why a violation was accepted or deferred. The reliable approach is a scripted pipeline that decouples error extraction from spatial correction: arcpy materializes errors from the geodatabase, geopandas vectorizes them for fast filtering, and every actionable violation is exported with the OriginFeatureID and rule code attached so the remediation is traceable and compliance-defensible.

arcpy-to-geopandas topology error extraction and routing pipeline A left-to-right flow: arcpy ValidateTopology materializes error feature classes (model stage); a SearchCursor extracts RuleID, OriginFeatureID, IsException, and geometry (field-ingest stage); a geopandas GeoDataFrame collapses tolerance-clustered duplicate defects with a half-tolerance buffered self-join; the deduplicated set then splits — reviewed IsException records are preserved and never overwritten, while actionable violations are routed to a CMMS work-order queue carrying their OriginFeatureID and rule code. ValidateTopology materialize errors SearchCursor extract fields GeoDataFrame dedup clusters split IsException? RuleID · OriginFeatureID · IsException · Shape@ IsException = True Reviewed exceptions preserved — never overwritten False CMMS work orders actionable + rule code Extraction stays GDB-native (arcpy); filtering stays vectorized (geopandas); routing stays auditable.

Environment Prerequisites

This procedure assumes a schema-aware ArcGIS Utility Network environment. Confirm each precondition before running anything:

  • ArcGIS Pro 3.x with the Utility Network extension licensed and the arcpy site package on the active Python interpreter.
  • Python 3.9+ with geopandas (and its shapely/pyproj/fiona stack) installed in a conda environment that can also import arcpy — keep them in the same interpreter so the extraction and vectorization steps share one process.
  • A validated topology in a file or enterprise geodatabase with its rule set already reflecting current engineering standards; stale rules generate noise that drowns real defects.
  • Exclusive workspace access — no concurrent edit sessions or schema locks, which will interrupt ValidateTopology_management and corrupt error materialization.
  • A known cluster tolerance that matches the field survey accuracy of the source data (typically 0.1–0.5 m for distribution networks); mismatches are the dominant cause of false positives.
  • Read access to the rule-code-to-business-logic mapping for your network so numeric RuleID values can be translated into engineering directives.

Schema-Aware Validation Protocol

Run these diagnostic checks in order before the main automation. The most common failure cause — tolerance and projection drift — comes first, because it produces thousands of false positives that make every later step unreliable.

  1. Workspace and lock validation. Confirm exclusive access to the geodatabase. Active schema locks or concurrent editing will interrupt validation and leave the error feature classes half-written.
  2. Spatial reference alignment. Confirm the topology dataset, the error feature classes, and any reference baselines share an identical coordinate system and vertical datum. Mismatched projections introduce sub-meter drift that cascades into spurious overlap and coincidence violations — the same class of failure that disciplined CRS alignment and geodetic transformations is designed to eliminate.
  3. Rule activation and tolerance calibration. Review the active topology rule set in ArcGIS Pro and confirm that ArcGIS Pro’s cluster tolerance matches field capture accuracy. Disable legacy rules that no longer reflect current standards before materializing errors.
  4. Error materialization scope. Restrict validation to the affected extent with a bounding polygon or feature selection. Full-network validation on datasets above ~500,000 features adds unnecessary I/O during incident triage.
  5. Schema field inspection. Confirm the error feature class exposes RuleID, OriginFeatureID, IsException, and a usable Shape@ token, and that the IsException flags reflect genuine, formally reviewed acceptances rather than stale defaults — these drive whether a violation is routed for remediation. This baseline error inventory is what the broader topology validation and tracing workflows consume to guarantee that connectivity tracing, pressure-zone validation, and fault isolation run against verified geometry.

Minimal Reproducible Implementation

The following pattern materializes errors with arcpy, extracts the schema-relevant fields, loads them into a geopandas GeoDataFrame, removes tolerance-clustered duplicates, and exports an auditable list of actionable violations. Extraction stays GDB-native (via arcpy) and filtering stays vectorized (via geopandas), so neither side becomes an in-memory bottleneck. The code is copy-paste-ready, handles the common failure modes explicitly, and returns structured output.

import os
import logging
import arcpy
import geopandas as gpd

# Structured logging for audit trails
logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s | %(levelname)s | %(message)s")

# --- Configuration -----------------------------------------------------------
GDB_PATH = r"C:\UtilityNetwork\Distribution.gdb"
TOPO_NAME = "PrimaryNetwork_Topology"
ERROR_FC = os.path.join(GDB_PATH, f"{TOPO_NAME}_line")
OUTPUT_CSV = r"C:\UtilityNetwork\batch_topology_errors.csv"
CLUSTER_TOLERANCE = 0.25  # Meters — MUST match the topology configuration


def validate_and_extract_topology():
    """Materialize topology errors and return schema-aware actionable records."""
    if not arcpy.Exists(GDB_PATH):
        raise FileNotFoundError(f"Geodatabase not found: {GDB_PATH}")

    logging.info("Validating topology and materializing error features...")
    try:
        # Validate within the default extent; restrict to a polygon for triage.
        arcpy.ValidateTopology_management(os.path.join(GDB_PATH, TOPO_NAME))
    except arcpy.ExecuteError:
        logging.error("Validation failed — check locks, rule activation, tolerance.")
        raise

    if not arcpy.Exists(ERROR_FC):
        raise RuntimeError(f"Error feature class not generated: {ERROR_FC}")

    fields = ["RuleID", "OriginFeatureID", "IsException", "Shape@"]

    logging.info("Extracting error records via arcpy.da.SearchCursor...")
    records = []
    with arcpy.da.SearchCursor(ERROR_FC, fields) as cursor:
        for rule_id, origin_id, is_exception, shape in cursor:
            records.append({
                "RuleID": rule_id,
                "OriginFeatureID": origin_id,
                "IsException": bool(is_exception),
                "geometry": shape,
            })

    if not records:
        logging.info("No topology errors detected. Network is compliant.")
        return None

    crs = arcpy.Describe(GDB_PATH).spatialReference.exportToString()
    gdf = gpd.GeoDataFrame(records, geometry="geometry", crs=crs)

    # Drop pre-approved, formally reviewed exceptions — never overwrite them.
    actionable = gdf[~gdf["IsException"]].copy()
    logging.info("Extracted %d errors; %d require remediation.",
                 len(gdf), len(actionable))
    return actionable


def apply_spatial_filters(gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    """Collapse tolerance-clustered duplicate defects and tag a violation group."""
    if gdf.empty:
        return gdf

    # Buffer by half the cluster tolerance so defects within tolerance of each
    # other are treated as a single defect during the spatial self-join.
    buffered = gdf.copy()
    buffered["geometry"] = gdf.buffer(CLUSTER_TOLERANCE / 2)
    buffered = buffered.set_geometry("geometry")

    duplicates = buffered.sjoin(buffered, how="inner", predicate="intersects")
    unique_indices = duplicates.index.drop_duplicates()

    filtered = gdf.loc[unique_indices].copy()
    filtered["ViolationGroup"] = filtered["RuleID"].astype(str).str.zfill(3)
    return filtered


def main():
    try:
        errors = validate_and_extract_topology()
        if errors is not None and not errors.empty:
            cleaned = apply_spatial_filters(errors)
            cleaned.to_csv(OUTPUT_CSV, index=False)
            logging.info("Actionable errors exported to %s", OUTPUT_CSV)
            return cleaned
        return None
    except Exception as exc:  # noqa: BLE001 — surface, log, and re-raise for CI
        logging.critical("Batch processing failed: %s", exc)
        raise


if __name__ == "__main__":
    main()

Two details carry most of the reliability. First, RuleID is meaningless until it is mapped to engineering language — translate RuleID=12 into service lateral must not cross mainline and RuleID=4 into duplicate asset placement through a lookup dictionary so generated work orders are actionable rather than cryptic. Second, the half-tolerance buffer and spatial self-join in apply_spatial_filters are what suppress the GNSS-versus-tolerance false positives: when crews capture at ±0.5 m but the topology cluster tolerance is 0.1 m, legitimate overlaps trip violations, and deduplicating clustered geometries collapses those artifacts into one defect. The same flagging pattern generalizes to the disconnected-feature signatures handled in how to fix disconnected edges in utility topology.

Production Deployment Pattern

Promote this script from a desk run to an operational control by wrapping it in the same governance you would give any change to the network:

  • Versioned workspace isolation. Run extraction against a dedicated version or a topology snapshot, not the default editing version, so concurrent field edits cannot move geometry mid-validation and so the exported error list reflects a single consistent state.
  • CI/CD pipeline hook. Parameterize GDB_PATH, TOPO_NAME, and CLUSTER_TOLERANCE, then run the script as a gate that fails the build when actionable error counts exceed a threshold. This mirrors the schema-promotion gating in automating connectivity rule validation in CI pipelines and keeps regressions out of production before they reach a trace.
  • CMMS / REST integration. Instead of leaving a CSV on disk, post each actionable violation to a work-order endpoint with its OriginFeatureID, ViolationGroup, and a geometry export attached, so field dispatch and forensic audit share one record. Never clear or overwrite IsException records automatically — route, do not resolve.
  • Retry and backoff. Wrap ValidateTopology_management and any REST calls in bounded exponential-backoff retries; transient enterprise-geodatabase lock contention is normal at scale and should not fail the whole run.
  • Scaling. For multi-tenant or multi-jurisdiction estates, containerize the arcpy extraction layer and schedule it alongside the wider batch topology processing with Python workflows so validation runs unattended across distributed environments.

Conclusion

This procedure automates the full path from raw topology violation to an auditable, actionable work item: arcpy materializes and extracts errors, geopandas deduplicates tolerance artifacts, and reviewed exceptions are preserved rather than clobbered. Running it on a version-isolated snapshot inside a CI gate turns spatial integrity from a one-off migration cleanup into continuous, compliance-defensible monitoring with a traceable record for every accepted or remediated defect. Clean topology is the precondition for trustworthy tracing — the logical next step is to wire the exported error list into the connectivity and isolation workflows that depend on it, starting with upstream and downstream tracing.