Designing a single computation path for a cross-service metric

claim 1.1dependent — cites claim 1 (Dynamox)

Two services independently calculated the same customer-facing metric, causing inconsistent data. I redesigned the architecture so only one service owned the calculation while every other service simply signaled stale data.

impactRemoved race conditionsSimplified architectureImproved customer trust

Context

Dynamox's platform derives a completion metric for industrial inspection routes: how much of a route has been inspected, per asset, per customer. The metric is customer-facing, appearing on dashboards operators use to plan work.

The underlying facts live in events. Inspections, edits and deletions flow through Kafka into multiple services, each maintaining its own projection of the data.

Problem

Two services computed the metric independently, each from its own projection. Under normal conditions they agreed. Under retries, out-of-order delivery or partial failures, they diverged, and customers saw two different numbers for the same route depending on which screen they opened.

Each divergence became a support ticket, a manual reconciliation, and a small withdrawal from customer trust. The team had built a reconciliation job to patch differences, which treated the symptom and added a third component that could disagree.

Constraints

—At-least-once delivery: every consumer must tolerate duplicates and reordering.

—No distributed transactions, since services deploy and fail independently.

—Zero downtime: the metric is in daily operational use.

—Historical data had to be backfilled to a consistent state.

—The two computing services were owned by different people; any fix had to survive team boundaries.

Alternatives Considered

—Keep both computations, improve the reconciliation job. Rejected: reconciling two independent computations is unbounded work, because every new edge case reappears twice.

—Extract the calculation into a shared library. Rejected: identical code over non-identical projections still diverges. The bug was in the data, not the formula.

—Compute on read at the API gateway. Rejected: pushed latency onto every dashboard load and still required a consistent source projection.

—Single owner service; all others emit staleness signals. Chosen.

Decision

Exactly one service owns the metric. It is the only code path in the company allowed to compute it.

Every other service that touches underlying data stops computing anything. Instead it emits a lightweight 'stale' signal: 'route X may have changed'. The owner recomputes the metric from the source of truth, idempotently, whenever a signal arrives.

This is where a personal principle crystallized: eventually consistent derived data should have exactly one computation path. And its corollary: if a value can always be recomputed from source, favor recomputation over synchronization.

Trade-offs

—Recomputation costs more than incremental updates. Accepted: the computation is cheap relative to the cost of divergence, and signals are debounced.

—A staleness window exists between signal and recomputation. Accepted: seconds of staleness with guaranteed convergence beats instant values that can be permanently wrong.

—The owner service becomes a critical path. Mitigated: idempotent consumers, dead-letter queue, and alerting on signal lag.

Implementation

Staleness signals travel over a dedicated Kafka topic keyed by route, so recomputations for the same route serialize naturally. The owner consumes with idempotent handlers, so recomputing twice is safe by construction and retries need no special handling.

A backfill script recomputed every historical metric from source, migrating the system to a consistent baseline before the new path went live. The old computation in the second service was deleted rather than disabled, because leaving it dormant invited resurrection.

Impact

—Inconsistency tickets for the metric dropped to zero after rollout.

—The reconciliation job was deleted, removing an entire class of maintenance.

—Race conditions became structurally impossible rather than merely unlikely.

—The pattern was reused for other derived data; I became the team's reference for cross-service synchronization.

Lessons Learned

Consistency debates end when ownership is explicit. Most of the design work was getting agreement on the sentence 'only this service computes this value', not writing code.

Deleting code is part of the architecture. The migration wasn't done until the second computation path physically ceased to exist.

Evidence

This document is the primary evidence for the capabilities it claims: Distributed Systems (event-driven consistency design), Architecture (ownership boundaries), Ownership (proposal through backfill through deletion), Communication (the decision survived because it was written down and agreed across two service owners).