Responding to a production incident with the missing architecture decision, not just a patch

at Dynamox

Root-caused a customer-blocking production incident to an undocumented architecture assumption, rebuilt the affected consumer with a post-mortem, an ADR, and business-rules documentation, then, after rollout exposed a database deadlock, diagnosed it to a Kafka partitioning mismatch and replaced silent message loss with retry, a dead-letter queue, and per-user serialization.

impactResolved the customer-blocking incident and documented the missing architecture decision so the same gap can't reopen silently.Gave the domain a properly workspace-aware consumer in the team's current service, where none had existed before.Eliminated the production deadlock at its root cause (partition-key mismatch), not by papering over the symptom.Replaced silent message loss with a recoverable failure path, with retry for transient errors and a dead-letter queue for permanent ones, closing a gap where failures had previously disappeared without a trace.Qualitative: higher confidence in the reliability of a permissions-critical consumer; no incident of the same kind recurred after the fix.

Context

This is my clearest evidence of incident response that fixes the class of bug rather than the instance, and of staying with a problem across two acts: the initial incident, and the subtler failure mode that only appeared once the fix was in production. Both times, I resisted the fastest available patch in favor of understanding why the system had been wrong in the first place, and left the answer documented so the next person wouldn't have to rediscover it.

It also shows an operational maturity that goes beyond "fixed the bug": diagnosing a production deadlock from logs, tracing it to a specific mismatch between a messaging system's partitioning and the data's actual contention pattern, is systems-level reasoning under pressure, the kind of debugging that separates "restarted the pod" from actually understanding the failure.

Problem

A customer was stuck on a permanent "sync error" screen and could not complete their work, a hard block rather than a cosmetic bug. The proximate cause traced to how a permissions-management consumer handled an edit: a decision about whether that consumer's logic should be scoped to a workspace had never been made explicit anywhere, so a change that assumed the wrong scope went out uncaught. The underlying consumer for this domain also only existed in the team's legacy service; the newer service had no equivalent, which was part of why the gap had never surfaced before.

Months after the rebuild shipped, a second, unrelated-looking problem appeared: the new consumer started throwing database deadlocks in production.

Constraints

—The real defect wasn't in the code path that failed. It was in a decision that was never written down. Patching the immediate scope bug would have left the same class of mistake possible on the next change, because nothing recorded *why* the consumer needed to behave the way it should.

—The user's own words made the stakes concrete. They were blocked mid-task and frustrated; there was no ambiguity about whether this mattered.

—The deadlock was intermittent and non-obvious. A generic "transaction failed" error gives no hint by itself that the actual cause is a mismatch between how work is distributed (partitioning) and how the underlying data is actually contended.

—The existing failure handling made the deadlock worse than it looked. Failed messages were being silently swallowed and their offset committed anyway, so before it could even be fixed properly, the failure mode itself had to change from "disappears without a trace" to "visible and recoverable."

Decision

I treated the incident as two separate diagnosis problems, months apart, and refused to close either with a surface-level fix.

—Wrote the post-mortem first. Before rebuilding anything, I documented what happened and why, so the incident had an owned record rather than just a fixed ticket.

—Made the missing decision explicit in an ADR. It covers why the consumer needs to respect workspace scope, how it should handle deletions, how to protect record accountability, and how edge cases should behave, instead of encoding the fix only in code where the next person would have to reverse-engineer the reasoning.

—Documented the business rules separately, because they had been scattered across application code with no central reference, which was itself part of why the original gap went unnoticed.

—Rebuilt the consumer properly in the team's current service and stack, rather than patching the legacy implementation, since the domain didn't have an equivalent there yet.

—Months later, diagnosed the deadlock from production logs, not guesswork. I searched for the specific error signature, found repeated bursts, and confirmed the mechanism: the topic was partitioned by the identifier of the change itself, not by the user it affected, so multiple messages about the *same* user could land on different partitions and be processed concurrently, colliding when they updated the same row.

—Fixed the mechanism, not just the symptom. I serialized updates per affected user so concurrent messages about the same person can no longer race each other, and replaced the silent failure path with retry for transient errors and a dead-letter queue for permanent ones, so a failure is now visible and recoverable instead of invisible.

Trade-offs

—Writing a post-mortem, an ADR, and business-rules documentation over shipping a direct fix. Documentation took real time the fastest patch wouldn't have, but the original gap existed *because* the decision was never written down, and repeating that mistake would have cost more later than it saved now.

—Rebuilding the consumer in the current stack over patching the legacy one. A patch would have been faster, but it would have kept the domain split across two services with no single source of truth, and left the newer service without behavior it needed.

—Diagnosing the deadlock's root cause over adding a retry and calling it fixed. A blind retry would have masked the collision without removing it; tracing the mismatch to the partition key made the fix address the actual contention instead of hiding it.

—Serializing per user over widening the transaction or the retry budget. Narrowing the fix to exactly the colliding scope (same user, concurrent messages) avoided a broader, vaguer slowdown that a more defensive fix would have introduced everywhere.

Impact

—Resolved the customer-blocking incident and documented the missing architecture decision so the same gap can't reopen silently.

—Gave the domain a properly workspace-aware consumer in the team's current service, where none had existed before.

—Eliminated the production deadlock at its root cause (partition-key mismatch), not by papering over the symptom.

—Replaced silent message loss with a recoverable failure path, with retry for transient errors and a dead-letter queue for permanent ones, closing a gap where failures had previously disappeared without a trace.

—Qualitative: higher confidence in the reliability of a permissions-critical consumer; no incident of the same kind recurred after the fix.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—An incident caused by an undocumented decision isn't fixed until the decision is written down. Otherwise you've fixed the symptom and left the cause free to resurface in a different shape.

—A generic "transaction failed" error is a starting point, not a diagnosis. The real cause is often a mismatch one layer up, here between how work was partitioned and how the data was actually contended.

—Never let a failure disappear silently. A system that swallows an error and commits anyway is worse than one that fails loudly, because you have to be able to see a failure before you can fix its cause.

—Fix the exact scope of the contention, not the whole surface around it. Serializing per affected user solved the actual collision without slowing down everything else.

Evidence

—Wrote the post-mortem, the ADR, and the business-rules documentation for the domain, ahead of rebuilding the consumer.

—Rebuilt the consumer in the team's current service and stack; activated in production.

—Independently diagnosed a later production deadlock to a Kafka partition-key mismatch via log analysis, and replaced silent message loss with retry, a dead-letter queue, and per-user serialization.

—Verified against the tracker. The incident is dated 2026-01-13, and the response was decomposed the same week: the post-mortem (2026-01-13 to 2026-01-19), the domain business-rules document (closed 2026-01-29) and the ADR (2026-01-19 to 2026-01-28) were all tracked tasks that *preceded* the rebuild, not write-ups produced afterwards.

—Verified rebuild: base service and messaging integration (2026-01-21 to 2026-01-26), the create and delete use cases (closed 2026-02-17), a bulk soft-delete endpoint for the affected relations, a feature flag in staging (2026-02-17), production activation (2026-03-16) with a production error investigated and closed the same day, and a separate task to repair the users the original bug had already corrupted (2026-01-16 to 2026-02-20).

—Verified follow-up, months later: a data-contract validation exception (2026-06-29), the transaction deadlock (2026-06-30 to 2026-07-01, four pull requests), a user-lifecycle inconsistency between upsert and role deletion (2026-07-07), and handling for the case where all of a user's roles are deleted at once.

—Source (private): Jira incident record and its linked post-mortem, ADR, and follow-up tasks, Dynamox engineering tracker; the corresponding pull requests.