8 min read1,796 words

Parallelising Topology Validation Across Tiles with Python

A full topology validation on an enterprise-scale utility network is measured in hours, which is why most estates stop doing it and rely on incremental validation of dirty areas alone. That works until something changes the graph outside a dirty area — a schema change, a rule change, a bulk import that suppressed dirty-area generation — and the full validation that would have caught it has not run since the spring. Parallelising the run across tiles brings it back into a maintenance window. The two things that make it correct rather than merely fast are buffering the tiles wider than the connectivity tolerance, so an error at a seam is seen rather than lost, and measuring the point at which more workers stop helping. This guide builds that runner, extending the batch patterns in batch topology processing with Python.

Environment Prerequisites

  • ArcGIS Pro 3.2+ with a Standard or Advanced licence and arcpy importable, plus enough concurrent licences and database connections for the worker count.
  • Python 3.11 with concurrent.futures from the standard library; the parallelism is process based because each worker needs its own arcpy state.
  • A partition source — a tile grid, a feeder layer, or a pressure-zone layer — with a feature count per partition so tiles can be balanced by work rather than by area.
  • A buffer distance greater than the network’s connectivity tolerance, so a seam is inside two tiles rather than between them.
  • A reconciled, isolated version or snapshot per worker, so validation does not contend with editors.
  • A findings store keyed by feature identifier, because de-duplication across seams depends on the key rather than on the tile.

Schema-Aware Validation Protocol — Run Before Distributing Work

  1. Balance tiles by feature count, not by area. A regular grid over a service territory produces one tile containing the downtown network and forty containing fields.
  2. Confirm the buffer exceeds the connectivity tolerance. A buffer smaller than the tolerance reproduces the seam problem it was meant to solve.
  3. Check the connection budget. Each worker holds a geodatabase connection; a worker count above what the database allows produces failures that look like validation errors.
  4. Verify the version or snapshot is stable for the run. A partition validated against a moving target produces findings that cannot be reproduced.
  5. Measure one tile before running all of them. The largest tile’s runtime multiplied by the queue depth is the wall clock, and it is worth knowing before committing a maintenance window.

Minimal Reproducible Implementation

from __future__ import annotations

import logging
from concurrent.futures import ProcessPoolExecutor, as_completed
from dataclasses import dataclass, field

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("tile-validate")


@dataclass
class TileResult:
    tile_id: str
    findings: dict[str, str] = field(default_factory=dict)   # feature id -> error code
    seconds: float = 0.0
    error: str = ""


@dataclass
class RunReport:
    tiles: list[TileResult] = field(default_factory=list)
    merged: dict[str, str] = field(default_factory=dict)

    @property
    def ok(self) -> bool:
        return all(not t.error for t in self.tiles)

    @property
    def slowest(self) -> TileResult | None:
        done = [t for t in self.tiles if not t.error]
        return max(done, key=lambda t: t.seconds) if done else None


def validate_tile(args: tuple[str, str, str, float]) -> TileResult:
    """Validate one buffered tile in its own process, with its own arcpy state."""
    import time

    import arcpy

    tile_id, un_path, extent_wkt, buffer_m = args
    result = TileResult(tile_id=tile_id)
    started = time.monotonic()
    try:
        envelope = arcpy.FromWKT(extent_wkt).buffer(buffer_m).extent
        arcpy.un.ValidateNetworkTopology(in_utility_network=un_path, extent=envelope)
        with arcpy.da.SearchCursor(f"{un_path}_Errors",
                                   ["FEATUREGLOBALID", "ERRORCODE"]) as cursor:
            for feature_id, code in cursor:
                result.findings[str(feature_id)] = str(code)
    except Exception as exc:                      # a worker must never take the run down
        result.error = f"{type(exc).__name__}: {exc}"
    result.seconds = time.monotonic() - started
    return result


def run_parallel(un_path: str, tiles: dict[str, str], buffer_m: float,
                 workers: int = 8) -> RunReport:
    """Validate every tile concurrently and merge the findings by feature id.

    ``tiles`` maps a tile id to its extent as well-known text. Findings are keyed by
    feature, so an error found in two overlapping tiles collapses to one entry rather
    than being reported twice.
    """
    report = RunReport()
    payloads = [(tid, un_path, wkt, buffer_m) for tid, wkt in tiles.items()]

    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(validate_tile, p): p[0] for p in payloads}
        for future in as_completed(futures):
            result = future.result()
            report.tiles.append(result)
            if result.error:
                LOG.error("tile %s failed: %s", result.tile_id, result.error)
            else:
                report.merged.update(result.findings)
                LOG.info("tile %s: %d finding(s) in %.1fs", result.tile_id,
                         len(result.findings), result.seconds)

    slow = report.slowest
    if slow:
        LOG.info("slowest tile %s at %.1fs — rebalance if it dominates",
                 slow.tile_id, slow.seconds)
    return report
Wall-clock time for a full validation as tile workers are added Validation parallelises well up to the point where the geodatabase becomes the constraint rather than the client. Beyond that, adding workers increases lock contention and the total time rises again. The useful number is not the theoretical maximum but the knee, which has to be measured on the estate rather than assumed from core count. full-extent validation, one estate, measured not assumed 1 worker 214 min serial baseline 4 workers 63 min near-linear 8 workers 41 min the knee 16 workers 58 min contention dominates Past the knee, more workers means more waiting; measure it before choosing.

The exception handler that catches everything inside the worker is deliberate. A single tile failing on a lock should reduce coverage by one tile and be reported, not terminate a run that has already spent thirty minutes on the other thirty-nine.

Why a tile boundary needs a buffer, and what happens without one A validation tile that stops exactly at its boundary cannot see the feature on the other side, so a connectivity error spanning the seam is invisible to both tiles. Buffering each tile by more than the connectivity tolerance means the seam is inside both tiles, the error is found twice, and the duplicate is removed afterwards — which is far cheaper than missing it. Tile A Tile B Main Main Gap at the seam seen by neither tile tile extent error at the seam Buffer wider than the connectivity tolerance, then de-duplicate.

Production Deployment Pattern

  1. Measure the knee, then set the worker count below it. The optimum is a property of the database and the network, not of the client machine, and it changes as the estate grows.
  2. Rebalance from the per-tile timings. A partition whose slowest tile takes four times the median is wasting most of the parallelism; split that tile next run.
  3. Run full validation on a schedule the estate can actually keep. Monthly and completed beats nightly and abandoned.
  4. Compare findings against the previous full run. New findings outside any dirty area are the ones this exercise exists to surface, and they usually indicate a rule or schema change.
  5. Keep incremental validation running as well. Full validation is a safety net, not a replacement for validating what was just edited.
  6. Persist the run. Tile definitions, buffer, worker count, per-tile timings and merged findings, so the next run is a comparison rather than a fresh start.
  7. Fail the run on tile failures, not on findings. Findings are the output; a tile that could not be validated is missing coverage, and reporting a clean result over an incomplete run is the one outcome worse than not running at all.
The parallel validation run, from partition to a single merged result The extent is partitioned into buffered tiles sized so that the largest is still comfortable for one worker. Each worker validates its tile independently against its own connection, writing findings keyed by feature. Findings are merged and de-duplicated across seams, and the run reports per-tile timings so the partition can be rebalanced next time. PARTITION buffered tiles balanced by count DISTRIBUTE one connection per worker VALIDATE independent per tile MERGE de-duplicate across seams REPORT per-tile timings rebalance next run Per-tile timings are what turn a partition from a guess into a measurement.

Conclusion

Parallel tile validation turns a job nobody runs into one that fits a maintenance window. Buffering tiles wider than the connectivity tolerance keeps seam errors visible; keying findings by feature makes the resulting duplication harmless; measuring the knee stops the worker count from becoming the bottleneck. What the run is really for is the class of defect incremental validation cannot see — and the comparison against the previous full run is where that class becomes visible.

For authoritative reference, consult the Python concurrent.futures documentation and the ArcGIS Pro topology validation reference.

Related pages