Correcting production data safely with reversible, auditable tooling

claim 1.5dependent — cites claim 1 (Dynamox)

Corrected tens of thousands of corrupted production records (~33k in one case) with no maintenance window, using reusable CLI commands with dry-run, rollback files, batch auditing and event re-publishing, a "correction command" pattern the team then reused.

impact~33,000 records corrected in production with no maintenance window, with an audit trail and a rollback available.A reusable "correction command" pattern adopted by the team (several commands built on it), turning a risky one-off into a repeatable, safe operation.Downstream consumers reconverged on corrected data via event re-publishing, so the fixes were complete end to end rather than source-only.Qualitative: higher confidence in production data integrity; corrections became routine and safe rather than exceptional and risky.

Context

This is my strongest evidence of operational maturity and ownership of data integrity. A production data fix is one of the higher-stakes things an engineer does: it's a write to live customer data with no undo unless you build one. Treating it with the rigor of a feature, meaning reversible, auditable, and previewable, rather than as a hurried script, is the distinction I want this to show.

It also shows I optimize for the team, not just the incident: I turned each fix into reusable tooling, so a one-off correction became a capability others could apply safely.

Problem

Several distinct bugs had left bad data in production, for example:

—~33,000 records with a timestamp stored without timezone information, so a time-sensitive field was wrong.

—Duplicate rows in a relationship table.

—Records polluted by an external integration's test run.

—Incorrect ownership assignments on records that should have had a single responsible party.

Constraints

—Writing to live production data has no natural undo. If a correction is wrong, you've now corrupted data twice, so reversibility has to be engineered in rather than assumed.

—No maintenance window. The system stayed live, so corrections had to be safe to run against a moving target and considerate of load.

—Downstream consumers had already ingested the bad data. Fixing the source record isn't enough if event-driven consumers still hold the old value. They have to be reconverged.

—Some corrections were subtle. A timezone fix has to be daylight-saving-aware, de-duplication has to pick the right survivor, and large backfills have to be paced so they don't overwhelm the system.

Decision

I refused to treat these as throwaway scripts and instead built them as tools with safety built in.

—Dry-run by default. Every command first reports exactly what it *would* change, so the effect is reviewed before anything is written.

—A rollback file written before any write. Each command records the prior state to a file up front, and a dedicated rollback command can restore it, so every correction is reversible.

—Batch auditing. Corrections run in audited batches, leaving a trail of what changed and when.

—Enriched event re-publishing. After fixing a source record, the command republishes the corresponding events so event-driven consumers reconverge on the corrected data rather than keeping the old value.

—Correctness in the specifics. The timezone fix was daylight-saving-aware, de-duplication chose survivors deterministically, and a large backfill used cursor pagination and paced micro-batches with bounded concurrency so it didn't overwhelm the system.

—Ran them against production carefully, diagnosing the containerized-environment quirks of executing inside live pods.

Trade-offs

—Dry-run and rollback file over a direct fix. Building preview and undo into every command costs real up-front effort, but a production data write without an undo is a bet I'm not willing to make. Accepted cost: each command is more work than a raw query.

—Republishing events over fixing only the source rows. Re-emitting events adds a step and some load, but without it the downstream consumers stay wrong and the fix is only half done.

—Reusable commands over one-off scripts. Generalizing into a "correction command" pattern cost more than a script for the one case, and paid back immediately as the same safety scaffolding was reused for the next correction.

—Paced batches over a single bulk write. Cursor pagination and micro-batches are slower than one big statement but keep a live system healthy during a large backfill.

Impact

—~33,000 records corrected in production with no maintenance window, with an audit trail and a rollback available.

—A reusable "correction command" pattern adopted by the team (several commands built on it), turning a risky one-off into a repeatable, safe operation.

—Downstream consumers reconverged on corrected data via event re-publishing, so the fixes were complete end to end rather than source-only.

—Qualitative: higher confidence in production data integrity; corrections became routine and safe rather than exceptional and risky.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—A production data fix deserves the rigor of a feature. Reversibility, auditability, and a preview are not optional extras when you're writing to live customer data.

—Dry-run by default; write the rollback before the write. The undo has to exist before the change, not after you discover you needed it.

—Fixing the source isn't enough in an event-driven system. Republish so downstream consumers reconverge, or you've fixed half the system.

—Turn a fix into tooling. Generalizing a correction into a reusable command turns a one-time risk into a capability the whole team can apply safely.

Evidence

—~33k records corrected in production without a maintenance window; audit trail and rollback available.

—Correction-command pattern (dry-run, rollback file, event re-publishing) reused across several team commands.

—Verified against the tracker, five distinct correction commands rather than one: routes left with more than one accountable user (2025-10-15), routes corrupted by an integration test against a partner system (2025-12-03 to 12-08), replaying affected routes through the new partial-update path (2026-01), users corrupted by a consumer bug (2026-01-16 to 2026-02-20), and duplicated cycles removed (2026-03-19 to 03-25). That spread across four months and four distinct data faults is what makes it a *pattern* rather than a one-off script.

—Source: Jira tasks in the inspection domain, 2025-10 to 2026-03, and the corresponding pull requests; record counts and the rollback/dry-run details from the consolidated career knowledge base (private).