10 min read2,274 words

Validating Field-Collected Assets Before Sync

The most expensive place to discover that a field observation is wrong is in the enterprise geodatabase, weeks later, when a trace behaves oddly. The cheapest is on the device, while the crew is still standing in front of the asset. Everything in between is a matter of pushing each check as early as it can meaningfully run — and accepting that a mobile client working from a cached schema will always let through values the server must reject. This guide builds the staging validator that sits between synchronisation and the network: it checks structure, live domains, positional accuracy against the asset class, and relationships, and it returns a disposition for every single edit rather than a verdict for the batch. It implements the validation layering set out in field data capture and mobile sync.

Environment Prerequisites

  • Python 3.11 with pandas>=2.0 and geopandas>=1.0 for the batch checks, and arcpy only for the relationship checks that need the network.
  • A staging area — a table or a file geodatabase — that receives synchronised edits before anything reaches a version.
  • The live coded-value domains, read at validation time rather than cached, so a retired value is caught here and not in the version.
  • A positional accuracy budget per asset class, expressed as a table the validator reads, and a reported accuracy stored on every captured point.
  • The crew and device identifiers on every edit, because a disposition has to be returned to someone specific.
  • A route into the work-management application for the disposition, since a portal nobody opens is the same as no feedback at all.

Schema-Aware Validation Protocol — Run Before the Validator

  1. Confirm the batch carries its capture metadata. Reported accuracy, capture method, crew, device and the replica check-out moment. A batch missing these can be validated structurally and not much else.
  2. Read the domains fresh. The whole point of staging validation is to catch what a cached list allowed; reading a cached list here reproduces the defect.
  3. Check the replica age against policy. Edits from a replica older than the limit should be escalated as a batch rather than rejected one by one, because they will all conflict.
  4. Confirm the accuracy budget table covers every class in the batch. A class with no budget defaults to accepting anything, which is the silent way an estate’s accuracy claim decays.
  5. Verify identifier uniqueness within the batch before comparing against the network. Two edits creating the same asset in one shift is a client or workflow defect, and it is cheaper to catch in the batch than after one of them is applied.

Minimal Reproducible Implementation

The validator below runs the four layers in order, short-circuits per edit on the first failure, and returns a disposition for every edit with the reason attached. It touches the network only for the relationship layer.

from __future__ import annotations

import logging
from dataclasses import dataclass, field

import geopandas as gpd
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("field-validate")

ACCEPTED, REJECTED, HELD = "ACCEPTED", "REJECTED", "HELD"


@dataclass
class Disposition:
    edit_id: str
    status: str
    reason: str = ""
    field_name: str = ""


@dataclass
class BatchResult:
    dispositions: list[Disposition] = field(default_factory=list)
    crew: str = ""
    replica_age_h: float = 0.0

    @property
    def accepted(self) -> list[Disposition]:
        return [d for d in self.dispositions if d.status == ACCEPTED]

    @property
    def ok(self) -> bool:
        return all(d.status == ACCEPTED for d in self.dispositions)


def validate_batch(
    edits: gpd.GeoDataFrame,
    domains: dict[str, set[str]],
    accuracy_budget: dict[str, float],
    required: dict[str, list[str]],
    known_parents: set[str],
) -> BatchResult:
    """Validate one synchronised batch and return a disposition per edit.

    ``domains`` and ``accuracy_budget`` are read live from the schema and the policy
    table. ``known_parents`` holds the identifiers of features a contained asset may
    legally attach to; an empty set disables the relationship layer for batches that
    carry no contained assets.
    """
    result = BatchResult(crew=str(edits["crew"].iloc[0]) if len(edits) else "")
    seen: set[str] = set()

    for _, row in edits.iterrows():
        edit_id = str(row["edit_id"])
        cls = str(row.get("asset_class", ""))

        # 1. structure
        missing = [f for f in required.get(cls, []) if pd.isna(row.get(f))]
        if missing:
            result.dispositions.append(Disposition(
                edit_id, REJECTED, f"required field(s) missing: {', '.join(missing)}",
                missing[0]))
            continue
        if row.geometry is None or row.geometry.is_empty:
            result.dispositions.append(Disposition(edit_id, REJECTED, "geometry is empty"))
            continue
        asset_id = str(row.get("asset_id", ""))
        if asset_id and asset_id in seen:
            result.dispositions.append(Disposition(
                edit_id, REJECTED, f"duplicate asset_id {asset_id} within the batch",
                "asset_id"))
            continue
        seen.add(asset_id)

        # 2. live domains
        bad_domain = None
        for fname, allowed in domains.items():
            value = row.get(fname)
            if value is not None and not pd.isna(value) and str(value) not in allowed:
                bad_domain = (fname, value)
                break
        if bad_domain:
            fname, value = bad_domain
            result.dispositions.append(Disposition(
                edit_id, REJECTED,
                f"{value!r} is not a current value for {fname} — the device list is stale",
                fname))
            continue

        # 3. accuracy against the class budget
        budget = accuracy_budget.get(cls)
        reported = row.get("reported_accuracy_m")
        if budget is None:
            result.dispositions.append(Disposition(
                edit_id, HELD, f"no accuracy budget defined for class {cls}"))
            continue
        if reported is None or pd.isna(reported):
            result.dispositions.append(Disposition(
                edit_id, REJECTED, "no reported accuracy captured with the position",
                "reported_accuracy_m"))
            continue
        if float(reported) > budget:
            result.dispositions.append(Disposition(
                edit_id, REJECTED,
                f"accuracy {float(reported):.2f} m exceeds the {budget:.2f} m budget "
                f"for {cls} — recapture with a corrected receiver",
                "reported_accuracy_m"))
            continue

        # 4. relationships
        parent = row.get("parent_id")
        if parent is not None and not pd.isna(parent) and str(parent) not in known_parents:
            result.dispositions.append(Disposition(
                edit_id, HELD, f"parent {parent} not found — needs an engineering decision",
                "parent_id"))
            continue

        result.dispositions.append(Disposition(edit_id, ACCEPTED))

    LOG.info("crew %s: %d accepted, %d rejected, %d held", result.crew,
             sum(d.status == ACCEPTED for d in result.dispositions),
             sum(d.status == REJECTED for d in result.dispositions),
             sum(d.status == HELD for d in result.dispositions))
    return result
What the staging validator checks, in the order that fails cheapest first Structural checks run first because they are free and they invalidate everything after them: required fields present, identifiers unique within the batch, geometry not null. Domain membership follows, checked against the live schema rather than the list the client cached. Positional accuracy is compared against the budget for the asset class, using the accuracy the receiver reported. Relationships are checked last because they are the only ones needing the network: does the parent exist, does the class permit this containment. Anything that fails is returned with the reason attached. STRUCTURE required fields unique ids DOMAINS against the live schema ACCURACY reported vs class budget RELATIONSHIPS parent exists containment legal DISPOSITION accepted, rejected or held the usual rejection CACHED DOMAIN LIST the client accepted a value the schema retired — a normal condition, not a fault Cheap checks first: a batch that fails structurally never reaches the network.

Two details carry most of the value. The rejection messages name the field and say what to do — “recapture with a corrected receiver” rather than “validation failed” — because the message is read by a crew, not by an engineer. And an unknown parent is held rather than rejected: the crew did nothing wrong, and the decision belongs to someone with the network in front of them.

What each validation layer can and cannot see The device has the crew and the asset in front of it and nothing else: no current schema, no network, no other crews. Staging has the full schema and the batch but not the topology. The version has everything, and it is also the most expensive place to discover a problem. Assigning each check to the earliest layer that can perform it is what keeps rejections cheap and correctable. Check Device Staging Version Required fields yes yes yes Live domain values cached only yes yes Accuracy vs class budget reported value yes yes Duplicate within batch no yes yes Parent exists no partly yes Connectivity legal no no yes Push every check to the leftmost column that can actually perform it.

Production Deployment Pattern

  1. Run the validator on every synchronisation, before a version exists. Creating a version for a batch that fails structurally wastes a version and complicates the audit trail.
  2. Return dispositions the same day. The value of a rejection decays quickly; a crew that is still in the area can recapture, and one that has moved on cannot.
  3. Escalate stale-replica batches as a batch. Rejecting forty edits individually because the replica was a month old teaches nothing; one message about the replica does.
  4. Track rejection reasons by crew and by class. A concentration in one crew is a training or equipment question; a concentration in one class usually means the budget or the domain is wrong rather than the capture.
  5. Never auto-correct a position. Snapping a captured point to the nearest main is tempting and destroys the only independent observation of where the asset actually is.
  6. Persist the batch result. The dispositions, the domains and budgets in force, and the replica age form the record that explains why an as-built record is missing an asset.
The disposition returned for every submitted edit, and where it lands The validator writes a disposition per edit rather than a verdict per batch, because a crew needs to know which of their twelve observations was rejected and why. Accepted edits move to a version. Rejected edits go back to the work-management application the crew already uses, carrying the asset, the field and the reason. Held edits — the ones needing an engineering decision — go to a review queue with a service level, and the crew is told they are held rather than left to assume they were accepted. Staging validator Version Crew application Review queue accepted edits, applied and posted rejected: asset, field, reason held: needs an engineering decision outcome returned when decided confirmation that it is live successes reported too, at summary level A per-edit disposition is what stops a crew assuming silence means success.

Conclusion

A staging validator turns synchronisation from a hopeful transfer into a checked handover. Running the layers in cost order keeps rejections cheap; reading domains live catches the class of defect a cached client list guarantees; comparing reported accuracy against a per-class budget is what keeps an estate’s accuracy claim supportable. Above all, returning a disposition per edit — with a reason a crew can act on — is what keeps field observation flowing into the model instead of into a notebook. The edits that pass still have to survive the office edits made while the crew was offline, which is the subject of offline mobile edits and conflict reconciliation.

For authoritative reference, consult the GeoPandas documentation and the pandas missing-data guide.

Related pages