The role-binding incident—and the separate deadlock found later
A role-cleanup change I implemented treated a user's access as global and removed it across workspaces. I owned the incident response, documented the missing rule, and led the architecture direction for a replacement consumer built by the squad. Months later, I traced a separate deadlock to Kafka partitioning and contributed per-user locking and recoverable failure handling; the later error rate was not measured.
Context
I changed a role-cleanup flow that removed a user's association after a permission change. I treated that association as global, although users can hold roles in multiple workspaces.
Problem
Two users lost access across workspaces and could not finish their work. The feature flag contained the incident that day, but the immediate mitigation did not explain or prevent the mistaken scope assumption.
Decision
I owned the post-mortem and wrote the architecture decision and business rules. The decision put deletion validation in the permissions-owning service; the consumer would replicate the result. I led the replacement epic and its architecture direction. The consumer implementation and rollout were shared squad work.
04Implementation
Months later, production logs showed a separate database deadlock in the replacement consumer. Messages about the same user could land on different Kafka partitions and run at once. I traced the mismatch and contributed a PostgreSQL advisory lock scoped to that user, plus retries and a dead-letter path so failures could be recovered instead of silently discarded.
Impact
The replacement consumer was activated in production in March 2026. The later locking and failure-handling changes followed in June and July. No post-deployment error-rate measurement is available, so I do not claim a measured reduction in deadlocks.