8 min read1,769 words

Caching Topology Snapshots for Degraded-Mode Operation

The moment a utility most needs its network model is the moment the model is most likely to be unreachable: a storm has taken out connectivity to the data centre, the enterprise geodatabase is saturated, or the trace service is queueing behind a bulk job. A degraded mode that returns a cached answer is therefore not a nicety — it is the difference between a dispatcher with an approximate isolation set and a dispatcher with nothing. What makes it safe is not the cache; it is the label. An answer that says “computed from a graph four hours old and device states from nine minutes ago” can be judged; the same answer unlabelled is indistinguishable from a live one and will be acted on as though it were. This guide builds that cache and that label, extending the tiered degradation model in fallback routing logic in legacy systems.

Environment Prerequisites

  • Python 3.11 with networkx>=3.0 for the cached graph, and a local store — an embedded database or a file cache — that survives a process restart.
  • A scheduled export of the network graph from a reconciled snapshot, following the isolation pattern in version versus snapshot isolation.
  • A device-state feed that can be cached independently of the graph, because it changes on a different timescale.
  • Per-layer freshness policy: the maximum age at which each cached layer may be used, and for which class of question.
  • A caller contract that carries the age label through to whoever reads the answer, including the user interface.
  • Timeouts on the live path short enough that failover happens before a dispatcher gives up.

Schema-Aware Validation Protocol — Run Before Relying on the Cache

  1. Confirm each layer records its own as-of time. A cache whose layers share a single timestamp cannot express the case that matters: a fresh graph with a stale device overlay.
  2. Check the cache survives a restart. A cache held only in memory disappears exactly when the process is restarted, which during an incident is likely.
  3. Verify the graph export succeeded recently. A failed nightly export leaves yesterday’s graph in place with no signal, so the export’s own health is part of the cache’s validity.
  4. Test the failover path deliberately. Failover that has never been exercised does not work; include it in the regular test cycle rather than discovering it during an event.
  5. Confirm the labels reach the user. A response that carries an age the interface discards provides none of the safety the design depends on.

Minimal Reproducible Implementation

from __future__ import annotations

import logging
import pickle
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from pathlib import Path

import networkx as nx

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
LOG = logging.getLogger("degraded-cache")

MAX_AGE = {
    "graph": timedelta(hours=12),
    "device_state": timedelta(minutes=30),
    "reference": timedelta(days=7),
}


@dataclass
class CachedLayer:
    name: str
    payload: object
    as_of: datetime

    @property
    def age(self) -> timedelta:
        return datetime.now(timezone.utc) - self.as_of

    @property
    def usable(self) -> bool:
        return self.age <= MAX_AGE.get(self.name, timedelta(hours=1))


@dataclass
class DegradedAnswer:
    result: object
    layer_ages: dict[str, float] = field(default_factory=dict)   # minutes
    safe_to_act: bool = False
    caveat: str = ""


class TopologyCache:
    """A restart-surviving cache of the layers a degraded answer is built from."""

    def __init__(self, directory: str) -> None:
        self.dir = Path(directory)
        self.dir.mkdir(parents=True, exist_ok=True)

    def put(self, layer: CachedLayer) -> None:
        with (self.dir / f"{layer.name}.pkl").open("wb") as fh:
            pickle.dump(layer, fh)
        LOG.info("cached %s as of %s", layer.name, layer.as_of.isoformat())

    def get(self, name: str) -> CachedLayer | None:
        path = self.dir / f"{name}.pkl"
        if not path.exists():
            return None
        with path.open("rb") as fh:
            return pickle.load(fh)

    def isolation(self, origin: str) -> DegradedAnswer:
        """Answer an isolation question from cache, labelled with what it was built from."""
        graph_layer = self.get("graph")
        state_layer = self.get("device_state")
        if graph_layer is None:
            return DegradedAnswer(result=None, caveat="no cached graph available")

        graph: nx.DiGraph = graph_layer.payload
        states: dict[str, str] = state_layer.payload if state_layer else {}

        reachable, frontier = set(), [origin]
        while frontier:
            node = frontier.pop()
            if node in reachable:
                continue
            reachable.add(node)
            if states.get(node, "OPEN") == "CLOSED":
                continue                      # a barrier stops the walk
            frontier.extend(graph.successors(node))

        ages = {"graph": graph_layer.age.total_seconds() / 60}
        if state_layer:
            ages["device_state"] = state_layer.age.total_seconds() / 60

        state_ok = bool(state_layer and state_layer.usable)
        caveat = ("device state is stale — plan only, do not switch"
                  if not state_ok else "")
        return DegradedAnswer(result=sorted(reachable), layer_ages=ages,
                              safe_to_act=state_ok and graph_layer.usable, caveat=caveat)
What each cached layer buys when the primary network is unreachable Degraded mode is not one cache but a stack of them, each with a different refresh cadence and a different consequence when it is stale. The graph changes slowly and can be hours old. Device state changes constantly and is the layer that decides whether an answer is safe to act on. Trace results precomputed isolation sets for common origins instant, may be stale Device state overlay open/closed per operable device stale in minutes Network graph nodes, edges, terminals, rules stale in hours Static reference asset attributes, criticality, customers stale in days Four cadences; only the second one makes an answer unsafe within minutes.

safe_to_act is deliberately conservative and deliberately separate from the result. The answer is still returned when the device overlay is stale, because knowing the approximate extent is useful; what changes is that the caller is told, in a field it cannot ignore, that this one must not drive a switching order.

What degraded mode may and may not be used for, by cache age A cached answer is not simply worse than a live one; it is usable for some questions and dangerous for others. The distinction is whether the answer will drive a physical action. Use Under 5 minutes Under 1 hour Older Locate an asset yes yes yes Estimate customers affected yes with a caveat no Plan an isolation yes, marked stale no no Execute a switching order no no no The bottom row is never yes: a switching order needs live device state.

Production Deployment Pattern

  1. Refresh each layer on its own schedule. The graph after each topology validation, device state every few minutes, reference data nightly. One schedule for all three wastes bandwidth and leaves the layer that matters stale.
  2. Fail over on a short timeout, not on an error. A live service that is slow is, during an incident, the same as one that is down.
  3. Surface the age in the interface. Not in a log, not in a header — where the dispatcher looks. The label is the whole safety mechanism.
  4. Exercise degraded mode on a schedule. A monthly drill that runs the dispatch console against the cache finds the defects that an incident would otherwise find for you.
  5. Cache per site, not centrally. A cache that lives in the same data centre as the service it backs up shares its failure modes.
  6. Log every degraded answer served, with the layer ages and whether it was acted on. That log is what justifies the investment and what shows where the freshness policy is wrong.
The failover path, and the label every degraded answer carries A request tries the live service first with a short timeout. On failure it falls back to the cache, and the response is labelled with the age of each layer it used. The caller decides whether that is acceptable for what it is about to do, which is only possible because the label exists. Caller Live service Cache Response request with a short timeout unavailable fall back answer + age per layer the label is the product caller decides if it is safe to act A degraded answer with no age label is indistinguishable from a live one.

Conclusion

Degraded mode is a labelling problem as much as a caching one. Splitting the cache into layers with their own refresh cadences lets the fast-moving one be judged separately from the slow ones; returning the age of every layer with the answer lets a caller decide what it may be used for; and refusing to mark an answer safe to act on when the device overlay is stale keeps the one dangerous use off the table. Built that way, a degraded answer is a useful, honest approximation rather than a live answer with a hidden expiry date.

For authoritative reference, consult the NetworkX documentation and the Python pickle module.

Related pages