Designing the safety guarantees for an AI-orchestrated cleanup, not just automating the grind

Designed and built a Claude Code skill that safely orchestrates parallel agents to remove expired feature flags across a shared frontend and two backend services, closing a 35-task cleanup epic (25+ flags plus obsolete product-tour infrastructure, ~3,100 lines removed in one commit) — with disjoint-file-set batching so agents can't collide, diff-against-baseline validation instead of absolute pass/fail, and an enforced two-repository deploy order so infrastructure changes can never precede the code that depends on them.

Context

This is my clearest evidence of treating AI agents as a workforce that needs real systems-design thinking, not just a faster way to type code. The interesting engineering here isn't "I used AI to remove some flags" — it's recognizing that running several agents concurrently on a shared codebase is a concurrency problem (agents can collide on the same file), that generic tooling output is noisy in a large, imperfect codebase (a raw typecheck or dead-code report drowns the real signal), and that one specific decision in this workflow can silently change what ships to production and therefore must stay a human's call. Encoding all three into a reusable tool, instead of doing the task once by hand and moving on, is the kind of leverage-building that scales past any single cleanup.

Problem

Feature flags and old product tours had accumulated across a shared frontend and two backend services, tracked as 35 individual removal tasks in one epic. The three services didn't share a convention for where a flag's value actually lives or how code reads it: the frontend keeps values in environment files with several code patterns for reading them (direct access, a typed helper, and local re-export indirections that had to be traced); one backend reads a typed environment service gated by a decorator; the other keeps its flag values in a separate infrastructure repository entirely, deployed independently from the service code that reads them.

That last point is the sharpest edge: removing a flag's value from the infrastructure repo before the corresponding code change has deployed can silently disable a feature that's still live in production, with no error to catch it.

Constraints

—No single mechanical rule covered all three repos. Each had a different storage convention and different code patterns for reading a flag, so a tool that worked for one repo's shape would silently miss cases in another.

—The flag's real production value decides what's safe to automate. A flag that's `true` in production means the old code path is genuinely dead and safe to delete automatically; a flag that's `false` might mean a feature the team still intends to ship — collapsing that distinction automatically risks deleting code nobody had abandoned.

—Parallelizing the removal introduces a new failure mode. Multiple agents editing a shared, large codebase at once can collide on the same file and corrupt each other's work if nothing coordinates who touches what.

—Standard safety checks are noisy at this scale. A monorepo with pre-existing typecheck errors and dead-code false positives makes an absolute pass/fail reading useless — it either hides a real regression in the noise or flags phantom ones.

—The two-repository case has an ordering invariant that's easy to get backward under time pressure, and getting it backward has a production consequence, not just a failed build.

Decision

I treated the parallel agents as a workforce that needed the same safety design any concurrent system does, and treated the one production-affecting decision as something the tool should surface, not resolve on its own.

—Encoded each repo's real storage and reading conventions explicitly, so classification is deterministic per repository instead of re-derived by hand each time.

—Made the flag's real production value drive the decision. `true` in production inlines to `true` and deletes the dead path automatically; `false` in production stops and asks, recommending the safe default (keep production behavior) rather than guessing silently — because that one branch can change what ships.

—Batched agents by disjoint file sets, computed from the overlap between the files each flag touches, so agents assigned to the same batch never write to the same file; flags that shared a file were serialized instead of parallelized.

—Validated by diffing against a pre-change baseline — typecheck error counts and a dead-code report, run once on the unmodified code and once after — instead of reading either tool's raw output, so only regressions the removal actually introduced show up.

—Made the two-repository deploy order an explicit, enforced step, not a note in a wiki page: the service's own change ships and deploys first; the infrastructure change that removes the now-unused variable is marked as dependent on that deploy and never allowed to precede it.

—Required an explicit confirmation gate before committing or opening a PR, with a checklist of affected screens or endpoints — derived from the files actually touched — for a human to verify manually before merge.

Trade-offs

—Disjoint-file-set batching over parallelizing by default. Computing file overlap before assigning work costs a planning step, but letting agents share a file without it risks a silently broken merge — worth the extra step once dozens of tasks are running concurrently.

—Diffing against a baseline over trusting a tool's raw output. Running each safety check twice costs time, but an absolute reading in a large, imperfect codebase either buries a real regression in pre-existing noise or reports phantom ones — neither is usable.

—Asking before inlining a flag that's `false` in production, over always deleting the path that never shipped. A slower, human-confirmed step here is the right cost, because automating it wrong means silently killing a feature the team hadn't abandoned.

—Enforcing the deploy order in the tool over documenting it and trusting reviewers. A rule that only lives in a person's memory doesn't survive turnover or a rushed release; encoding it means it can't be skipped by accident.

Impact

—Closed a 35-task cleanup epic spanning a shared frontend and two backend services, removing 25+ expired feature flags and obsolete product-tour infrastructure — one commit alone removed roughly 3,100 lines of dead code.

—Produced a reusable tool, not a one-off script: the same skill applies to the next flag-cleanup epic in any of the three repositories, encoding conventions that used to live only in memory.

—Removed the failure modes that make this kind of cleanup dangerous by hand — misclassifying a flag, agents colliding on a shared file, or getting the two-repository deploy order backward — by designing them out of the tool rather than depending on whoever runs it remembering all three under time pressure.

—Qualitative: a simpler, more legible codebase as a direct result of the dead-code removal; no isolated before/after build-time metric was captured for this change alone.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—Parallel AI agents need the same safety design as any other concurrent workers. Isolate their work by disjoint file ownership, or they will corrupt each other's changes exactly like any other race condition.

—Validate against a baseline, not an absolute reading, in any codebase old enough to have pre-existing noise. Otherwise the signal you actually care about is either buried or drowned by false positives.

—Encode an invariant in the tool; don't just document it. A rule that depends on a person remembering it under pressure will eventually be skipped — a tool that enforces it can't be.

—Automate everything except the one decision that can silently change production behavior. That single branch deserves a human, even inside an otherwise fully automated pipeline.

Evidence

—Closed Jira epic with 35 subtasks across a shared frontend and two backend services; 25+ feature flags and obsolete product-tour infrastructure removed, including one commit that removed roughly 3,100 lines of dead code.

—Built a reusable Claude Code skill encoding the classification, parallel-safety, and validation rules for all three repositories' conventions.

—Part of a broader, sustained practice of building reliable tooling on top of AI coding agents: a from-scratch MCP server exposing SonarCloud's API for quality-gate review inside the same workflow, and further tools automating bug-investigation-to-shipped-fix (with git/Jira archaeology and PR creation), team-scoped production-error triage, and post-incident latency analysis.

—Source (private): Dynamox engineering tracker (Jira epic and subtasks) and personal tooling repository.