Synchronizing a Cross-Service Data Edit Atomically

claim 1.2dependent — cites claim 1 (Dynamox)

Made an edit to an organizational node in the asset hierarchy propagate consistently across denormalized copies in seven tables and two services, via a single atomic transaction with post-commit events, the work that made me the team's reference for cross-service synchronization.

impactMade organizational-node edits propagate consistently across the seven-table, two-service surface, with no more contradictory data from a partial update.Became the team's reference for cross-service data synchronization (recognized in a performance review), and this work anchored a run of related synchronization epics.Removed a domain leak and a latent DI bug in the follow-up refactor, leaving the code more maintainable than the feature alone required.Qualitative: higher trust in customer-facing hierarchy data; no hard before/after number was captured.

Context

This is the experience that established me as the go-to person for cross-service data consistency on my team, a reputation I then carried across several related pieces of work. It is strong evidence that I can hold a consistency boundary in my head across services that don't share a transaction, decompose a large epic into tracked, reviewable work, and reason about event ordering rather than just "make the write happen".

It also taught me to see a bloated use case as a design smell: the follow-up refactor, where I pulled apart an oversized use case and uncovered a real dependency-injection bug, is part of why this changed how I think about domain boundaries.

Problem

The product lets users reorganize their asset hierarchy: rename an organizational node, edit its description, or move it to a different position in the tree. For performance, data about that node was denormalized, with copies and derived references living across roughly seven tables, split between two independently deployed backend services.

A single user edit therefore had to fan out into many coordinated writes across a service boundary. If only some of those writes landed, the product would display inconsistent data: a node named one thing here and another there, or attached to the wrong branch of the tree. There was no shared transaction spanning the two services to lean on.

Constraints

—The consistency boundary crossed a service boundary. Two services, deployed and scaled independently, had to agree on the result of one edit, with no distributed transaction available.

—Two distinct flows, different blast radius. A simple field edit (name/description) is contained; moving a node to a new path in the tree cascades into route context, teams, checklists and categories that reference it. The move case is where partial failure hurts most.

—Events could arrive stale or out of order. Because propagation was event-driven, a later edit's event could be overtaken by an earlier one, silently reverting the data.

—It was a large epic, not a single change. The work had to be decomposed into independently reviewable pieces without losing the end-to-end consistency guarantee.

Decision

I decomposed the epic into a sequence of tracked subtasks and treated the consistency guarantee, rather than the individual writes, as the thing I was actually building.

—Made each edit one atomic transaction. All the writes for a single edit succeed or fail together, so the denormalized copies can never be left half-updated. The two flows (simple edit vs. path move) shared this boundary but differed in scope, with the move flow recomputing the affected route context.

—Published events only after the transaction committed. Nothing downstream is told the edit happened until it durably has, so a rollback can't leak an event for a change that didn't stick.

—Discarded stale events on the consuming side. Consumers compare the edit's timestamp against what they already hold and drop anything older, so out-of-order delivery can't revert newer data.

—Shipped behind a feature flag per environment so the new path could be rolled out and rolled back safely.

—Refactored afterward, not before. Once it worked, I broke apart an oversized use case (it had grown to over a thousand lines and many dependencies, a domain leak) and, in doing so, found and fixed a real dependency-injection bug (a missing provider).

Trade-offs

—A single atomic transaction over a cascade of independent events. Coordinating the writes as one transaction guarantees all-or-nothing consistency; a cascade of events would have been looser and simpler to write but reintroduces exactly the partial-update problem. Accepted cost: a larger transaction and tighter coupling to the data store.

—Publishing events after commit over publishing them inline. Post-commit publishing means downstream systems never hear about a change that later rolled back, at the cost of a small window where the write is done but the event hasn't been sent yet.

—Discarding stale events by timestamp over trusting delivery order. Comparing timestamps costs an extra check per event but removes a whole class of "newer data silently reverted" bugs that message ordering alone can't prevent.

—A feature flag over a hard cutover. The flag added branching but made an irreversible-looking data change reversible in production.

Impact

—Made organizational-node edits propagate consistently across the seven-table, two-service surface, with no more contradictory data from a partial update.

—Became the team's reference for cross-service data synchronization (recognized in a performance review), and this work anchored a run of related synchronization epics.

—Removed a domain leak and a latent DI bug in the follow-up refactor, leaving the code more maintainable than the feature alone required.

—Qualitative: higher trust in customer-facing hierarchy data; no hard before/after number was captured.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—Denormalized data needs one atomic write boundary. If a single logical edit maps to many physical writes, they must succeed or fail together. Anything less will eventually show the user a contradiction.

—Publish events only after the transaction commits. Otherwise a rollback leaks a message about something that never happened, and downstream consumers diverge.

—Assume events arrive stale and out of order. A cheap timestamp comparison on the consumer is what keeps late deliveries from reverting newer state.

—A use case that keeps growing is a domain boundary asking to be drawn. Size and dependency count are design signals, not just cleanliness issues.

Evidence

—Owned end to end: epic decomposition into tracked subtasks, implementation of both flows, and rollout behind a feature flag.

—Follow-up refactor split an oversized use case and fixed a latent dependency-injection bug it had been hiding.

—Cited as the basis for becoming the team's cross-service-synchronization reference (performance review).

—Verified against the tracker (2026-03 to 2026-06): the story was opened 2026-03-12 and resolved 2026-06-12, with nine implementation subtasks executed 2026-04-22 to 2026-05-19, decomposed one-per-table plus one for the simple-edit path and one for recomputing route context.

—Verified surface: the seven denormalized tables named in the requirement are the organizational-node table, its route-scoped copy, checklists, routes, categories, teams and quizzes, with a downstream event published for checklists specifically.

—Verified guarantees, as written into the acceptance criteria before implementation: a single atomic transaction with rollback on error; idempotency by discarding events whose update timestamp is not newer than the stored one; audit records carrying old and new values for both the edit and the move case; and a gradual rollout behind a pre-existing feature flag that until then was wired to nothing.

—Verified follow-up: a refactor task (2026-06) split the oversized use case for create/update/delete, and a separate consumer fix stopped a machine move from removing measurement points from a route by discriminating on parent id.

—Behaviour documented as BDD scenarios in a dedicated task, per the ADR's own documentation requirement.

—Source (private): Jira story and subtasks in the inspection domain, 2026-03 to 2026-06; consolidated career knowledge base.