9 min read2,014 words

Calibrating Service Life Tables from Failure History

Every risk model in a utility rests on an assumption about how long each material lasts, and in most estates that assumption came from a handbook. Published service-life figures are averages over soils, water chemistry, installation practice and operating pressure that no single utility experiences — which means they are wrong locally in both directions. Cast iron in aggressive soil fails well before the published figure; the same material in benign ground outlasts it by decades. A replacement programme driven by the published number therefore over-invests in one population and under-invests in another, and the failure record that would have revealed this is usually sitting unused in the work-management system. This guide calibrates the table against that record, handling the censoring that makes the difference between a survival estimate and a misleading average. It feeds the risk model described in condition-based maintenance scheduling.

Environment Prerequisites

  • Python 3.11 with pandas>=2.0 and numpy>=1.26; a survival library is optional and the worked estimator below uses neither.
  • A failure history naming assets, not addresses. A failure record joined by location is a join you cannot trust; the record needs an asset identifier.
  • Install dates on the surviving population, because survivors are the larger half of the evidence and dropping them biases everything.
  • An environment grouping — soil corrosivity, pressure zone, installation era — since material alone rarely explains the spread.
  • A minimum-sample policy stating how many observed failures a group needs before its figure replaces the published one.
  • Version control for the resulting table, because it is a model input and a change to it changes every risk score in the estate.

Schema-Aware Validation Protocol — Run Before Fitting Anything

  1. Confirm failures resolve to assets that still exist. A failure on a retired and replaced main belongs to the retired asset, and attributing it to the replacement makes the new material look terrible.
  2. Check for repeat failures on one asset. A main that has failed four times is one asset with four events, and counting it as four assets distorts the sample.
  3. Include the survivors. Every asset still in service is a censored observation, and omitting them is the single largest source of bias in this exercise.
  4. Look for an install-date sentinel. A pile-up on 1900-01-01 produces implausible ages that will dominate any median.
  5. Confirm the environment grouping is populated. A material split by soil corrosivity is far more predictive than material alone, but only where the soil attribution actually exists.

Minimal Reproducible Implementation

from __future__ import annotations

import logging
from dataclasses import dataclass, field

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("service-life")

MIN_FAILURES = 30          # below this, the published figure stands


@dataclass
class LifeEstimate:
    group: str
    median_years: float | None
    failures: int
    survivors: int
    status: str            # CALIBRATED, PROVISIONAL, INSUFFICIENT
    note: str = ""


@dataclass
class Calibration:
    estimates: list[LifeEstimate] = field(default_factory=list)

    def table(self) -> dict[str, float]:
        return {e.group: e.median_years for e in self.estimates
                if e.median_years is not None and e.status == "CALIBRATED"}


def kaplan_meier_median(ages: np.ndarray, failed: np.ndarray) -> float | None:
    """Median survival time from right-censored observations.

    ``ages`` is the age of each asset at failure or at the observation date; ``failed``
    marks which of those ages ended in a failure. Survivors contribute to the risk set
    without contributing a failure, which is exactly what keeps the estimate from
    collapsing toward the age of the failures alone.
    """
    order = np.argsort(ages)
    ages, failed = ages[order], failed[order]
    at_risk = len(ages)
    survival = 1.0

    for i, age in enumerate(ages):
        if failed[i]:
            survival *= (at_risk - 1) / at_risk
            if survival <= 0.5:
                return float(age)
        at_risk -= 1
    return None            # never reached 50% survival within the observation window


def calibrate(assets: pd.DataFrame, failures: pd.DataFrame,
              as_of: pd.Timestamp) -> Calibration:
    """Estimate median service life per material-and-environment group."""
    result = Calibration()
    first_failure = (failures.sort_values("failed_at")
                     .groupby("asset_id")["failed_at"].first())

    joined = assets.copy()
    joined["failed_at"] = joined["asset_id"].map(first_failure)
    joined["end"] = joined["failed_at"].fillna(as_of)
    joined["age"] = (joined["end"] - joined["install_date"]).dt.days / 365.25
    joined["failed"] = joined["failed_at"].notna()
    joined = joined[joined["age"] > 0]

    for group, rows in joined.groupby("life_group"):
        n_fail = int(rows["failed"].sum())
        n_surv = int((~rows["failed"]).sum())
        if n_fail < 10:
            result.estimates.append(LifeEstimate(
                str(group), None, n_fail, n_surv, "INSUFFICIENT",
                "too few failures — keep the published figure"))
            continue

        median = kaplan_meier_median(rows["age"].to_numpy(),
                                     rows["failed"].to_numpy())
        status = "CALIBRATED" if n_fail >= MIN_FAILURES else "PROVISIONAL"
        note = "" if status == "CALIBRATED" else "provisional — sample below threshold"
        if median is None:
            status, note = "INSUFFICIENT", "survival never reached 50% in the window"
        result.estimates.append(LifeEstimate(str(group), median, n_fail, n_surv,
                                             status, note))

    LOG.info("%d group(s): %d calibrated", len(result.estimates),
             sum(e.status == "CALIBRATED" for e in result.estimates))
    return result
Published service life against observed life for the same materials in one estate Published service-life figures are national averages over soils, water chemistry and installation practices that no single estate experiences. Calibrating against local failure history usually moves them, sometimes substantially and not always in the same direction: cast iron in aggressive soil fails early, while ductile iron in benign conditions outlasts the book. Using the published number for both misallocates the replacement programme at both ends. median service life, one estate, thirty years of failure records Cast iron — published 100 yr handbook figure Cast iron — observed 74 yr aggressive soil Ductile iron — published 75 yr handbook figure Ductile iron — observed 92 yr benign conditions Calibration moves the numbers in both directions; only local data says which.

The censoring is the whole point of using a survival estimator rather than averaging the ages of failed assets. An estate whose cast iron is mostly still in the ground will show a median failure age far below its true service life if survivors are excluded — and the replacement programme built on that number will spend heavily on material that had decades left.

From failure records to a calibrated service-life table Failures are joined to the assets that produced them, which requires the failure record to name an asset rather than an address. Censoring is applied: assets still in service have not failed yet and must be included as survivors rather than dropped, or the estimate is biased toward early failure. A survival curve per material and environment gives a median life, and the table is only updated where the sample is large enough to support it. JOIN failures to assets by identifier CENSOR survivors counted not dropped FIT survival curve per material + environment UPDATE only where the sample supports it the classic error SURVIVORSHIP BIAS dropping assets that have not failed makes every material look short-lived Censoring is the step that separates a survival estimate from a failure average.

Using a Calibrated Table Responsibly

A calibrated figure is a better input, not a prediction about an individual asset. Three disciplines keep it from being over-read.

Publish the sample with the figure. A median supported by thirty-one failures and one supported by three hundred are different kinds of number, and a risk model that treats them identically inherits the difference invisibly.

Recalibrate on a schedule, not continuously. Service life moves slowly; a table that changes every week produces a risk ranking that churns for reasons nobody can explain to a regulator.

Keep the published figures as a fallback. Where a group is insufficient, the handbook number is the honest answer, and recording that it was used is what lets a reviewer see which parts of the model rest on local evidence.

What each sample size supports, and what it does not A calibrated figure is only as good as the failures behind it. Small samples produce confident nonsense, and the honest response is to widen the grouping rather than to publish a number with no support. The threshold is a policy decision that should be written down once and applied everywhere. Sample Failures observed Supports Under 10 anecdote keep the published figure 10 to 30 a direction adjust, flag as provisional 30 to 100 a median replace the published figure Over 100 a distribution model risk, not just life Widen the grouping rather than publishing a figure the sample cannot support.

Production Deployment Pattern

  1. Recalibrate annually against the full failure history, and treat the resulting table as a reviewed change rather than an automatic update.
  2. Version the table with the risk model. A risk score is only reproducible if the service-life table that produced it can be recovered.
  3. Group by material and environment, and widen the grouping rather than publishing a figure an insufficient sample cannot support.
  4. Report the coverage. The share of assets whose group is calibrated rather than published is the honest description of how locally grounded the model is.
  5. Feed repeat-failure assets into the model separately. An asset that has failed three times is telling you something the material average cannot, and it belongs in the intervention queue regardless of its age.
  6. Persist every calibration run — groups, samples, medians, status — because the next run is a comparison and a moving median is itself a signal.

Conclusion

Calibrating service life against local failure history replaces a national average with an estimate of what happens in this ground, to this material, under this operating regime. Including survivors as censored observations is what makes the estimate honest; a minimum-sample policy is what keeps it from producing confident nonsense on thin data; and publishing the sample alongside the figure is what lets the risk model downstream treat a well-supported number differently from a provisional one. The replacement programme that follows is then aimed by evidence rather than by handbook.

For authoritative reference, consult the NumPy statistics reference and the AWWA standards catalogue.

Related pages