8 min read1,717 words

Finding Orphaned Service Laterals with GeoPandas

A service lateral that is not connected to its main is a customer the network cannot find. It appears on every map, it is included in every asset count, and it is absent from every trace — which means absent from impact analysis during an outage, absent from an isolation boundary during a shutoff, and absent from the customer count in a regulatory filing. Laterals orphan in four distinct ways and only one of them is a snapping problem, so a repair queue that treats them all as gaps fixes a quarter of the population and quietly reports success. This guide builds the detection pass that separates the classes, using an exported snapshot and GeoPandas rather than a licensed engine, so it can run in continuous integration. It complements the repair workflow in network fragmentation and gap resolution.

Environment Prerequisites

  • Python 3.11 in an isolated environment with geopandas>=1.0, shapely>=2.0 and pandas>=2.0; no arcpy licence is needed for detection.
  • An export from a reconciled snapshot carrying laterals, mains, service points and the association table, all in one projected coordinate reference.
  • The network connectivity tolerance, which is what separates a geometric gap from a coincident endpoint.
  • Pressure zone or subnetwork attribution on both laterals and mains, so a lateral attached to the wrong main can be distinguished from one attached correctly.
  • A review queue for the classes that need an engineering decision rather than a repair.
  • A run store, so the counts by class can be compared between sweeps and the trend measured.

Schema-Aware Validation Protocol — Run Before the Sweep

  1. Confirm one projected coordinate reference across all three layers. Distance comparisons across mixed frames produce a gap count that measures the frames rather than the network.
  2. Check the association export is complete. A partial association table makes every lateral look orphaned, which is a spectacular false positive and easy to spot.
  3. Distinguish a lateral terminus from a lateral endpoint. The end that meets the customer is not supposed to touch a main, and counting it as an orphan doubles every finding.
  4. Verify pressure zone attribution exists on both sides. The wrong-parent class depends on it, and where it is missing that class is simply undetectable.
  5. Sample twenty findings by hand before trusting a sweep. The first run of any detector on a new estate finds mostly its own misconfiguration.

Minimal Reproducible Implementation

from __future__ import annotations

import logging
from collections import Counter
from dataclasses import dataclass, field

import geopandas as gpd
from shapely.geometry import Point

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("orphan-laterals")


@dataclass
class Orphan:
    lateral_id: str
    kind: str          # GAP, NO_ASSOCIATION, WRONG_PARENT, NO_SERVICE
    detail: str


@dataclass
class SweepResult:
    orphans: list[Orphan] = field(default_factory=list)
    checked: int = 0

    def by_kind(self) -> dict[str, int]:
        return dict(Counter(o.kind for o in self.orphans))


def find_orphans(
    laterals: gpd.GeoDataFrame,
    mains: gpd.GeoDataFrame,
    associations: set[tuple[str, str]],
    service_points: set[str],
    tolerance_m: float = 0.01,
) -> SweepResult:
    """Classify every lateral by how it fails to reach the network.

    ``associations`` holds (lateral_id, main_id) pairs that exist in the model. A
    lateral whose upstream endpoint is coincident with a main but which appears in no
    association pair is the class most estates never look for, and usually the largest.
    """
    result = SweepResult()
    sindex = mains.sindex

    for _, lat in laterals.iterrows():
        result.checked += 1
        lateral_id = str(lat["asset_id"])
        upstream = Point(lat.geometry.coords[0])

        # Which mains are near the upstream endpoint?
        candidates = mains.iloc[list(sindex.intersection(
            upstream.buffer(max(tolerance_m * 50, 1.0)).bounds))]
        distances = [(float(upstream.distance(m.geometry)), str(m["asset_id"]),
                      m.get("pressure_zone"))
                     for _, m in candidates.iterrows()]
        distances.sort()

        if not distances or distances[0][0] > tolerance_m:
            nearest = f"{distances[0][0]:.3f} m" if distances else "no main nearby"
            result.orphans.append(Orphan(lateral_id, "GAP",
                                         f"nearest main at {nearest}"))
            continue

        _, main_id, zone = distances[0]
        if (lateral_id, main_id) not in associations:
            result.orphans.append(Orphan(
                lateral_id, "NO_ASSOCIATION",
                f"coincident with {main_id} but no association row"))
            continue

        if zone is not None and lat.get("pressure_zone") not in (None, zone):
            result.orphans.append(Orphan(
                lateral_id, "WRONG_PARENT",
                f"lateral zone {lat.get('pressure_zone')} vs main zone {zone}"))
            continue

        if lateral_id not in service_points:
            result.orphans.append(Orphan(lateral_id, "NO_SERVICE",
                                         "no service point downstream of this lateral"))

    LOG.info("%d lateral(s) checked, orphans by kind: %s",
             result.checked, result.by_kind())
    return result
Three laterals that look identical on the map and are not The first lateral is connected: its endpoint is coincident with the main inside tolerance and an association exists. The second is geometrically coincident with no association, so it draws correctly and traces not at all. The third ends near the main but outside tolerance, which is a geometric gap. Only the first is a customer the network can find. association coincident only gap Distribution main Lateral A connected Lateral B no association Lateral C outside tolerance Service Service invisible to traces Service invisible to traces reachable orphaned Two of these three customers are absent from every impact count.

The order of the checks is the design. A lateral is tested for a geometric gap first because that invalidates everything after it, then for a missing association, then for a wrong parent, then for a missing service point. Each test is only meaningful once the previous one has passed.

Four orphan classes, how to detect each and what fixes it Orphaned laterals arrive in four distinct ways, and lumping them into one repair queue means three of the four get the wrong treatment. Only the first is a snapping problem. Class Detected by Fix Geometric gap endpoint distance snap within tolerance Coincident, no association association query create the association Wrong parent main pressure zone mismatch engineering review No service point no downstream point customer data, not GIS Only the first row is a snapping problem; the rest are modelling or data questions.

Production Deployment Pattern

  1. Run the sweep nightly against the reconciled snapshot and route by class: gaps to the repair pipeline, missing associations to a bulk fix, wrong parents to engineering review.
  2. Bulk-fix missing associations, carefully. Where geometry is coincident and the rule set permits the pair, creating the association is safe and can be done at volume — but write it in a version and validate before posting.
  3. Never bulk-fix a wrong parent. A lateral attached to the wrong main is usually a real modelling question about where the service is actually fed from.
  4. Track counts by class between runs. A rising NO_ASSOCIATION count points at an import that writes geometry without relationships, which is a pipeline defect rather than a data one.
  5. Cross-check against the customer system. A lateral with no service point may be a data gap on either side, and the reconciliation is what tells you which.
  6. Feed the confirmed orphan count into impact analysis. Until they are fixed, they are known customers absent from every trace, and a stated number is better than a silent one.
  7. Re-run the sweep after every bulk association fix. The fix and the detector disagree surprisingly often on the first attempt, usually because the rule set forbids a pair the geometry suggests, and finding that out in the same session is much cheaper than a week later.
Orphaned laterals found by class in a first sweep of one district The distribution is typical: most orphans are missing associations on geometrically perfect data, usually created by an import that wrote geometry without relationships. Geometric gaps are the minority, and they are the only class most estates look for. first sweep, one district of ~9,000 services Coincident, no association 1840 laterals import artefact Geometric gap 260 laterals snapping Wrong parent main 74 laterals needs review No service point 31 laterals customer data The class nobody looks for is usually the largest.

Conclusion

Orphaned laterals are the difference between an asset count and a customer count. Separating them into four classes turns an undifferentiated repair queue into three work streams and one engineering question, and it reveals that the largest class is usually the one nobody was looking for: geometrically perfect laterals with no association behind them. Running the sweep on a snapshot with GeoPandas keeps it cheap enough to run nightly, and the trend by class is what shows whether the pipeline that creates laterals has been fixed.

For authoritative reference, consult the GeoPandas documentation and the Shapely manual.

Related pages