Making a flaky end-to-end suite deterministic without hiding failures

claim 1.3dependent — cites claim 1 (Dynamox)

Turned a non-deterministic end-to-end suite (25 to 37 failures varying per run) into 302 passing with zero failures (8 pre-existing skips left untouched, and the consumer running), without adding a skip or weakening an assertion, and, in review, empirically proved that three of four proposed production changes were unnecessary.

impact302 tests passing, zero failures, deterministic behavior with the consumer running, from a suite that produced 25 to 37 varying failures per run. The 8 skips in the final count were pre-existing and left as they were; I added none.Backend CI became trustworthy again, so a red run once more means a real problem.Protected production code and business rules from being altered to appease a flaky test, by disproving three of four proposed changes in review.Qualitative: regressions are once again caught by the suite instead of hidden by it.

Context

This is my strongest evidence of methodological rigor and of technical leadership in code review. Anyone can make a flaky suite green by loosening it; the engineering is in refusing to. The constraints I set (no new skips, no weaker asserts, no bigger timeouts) are what forced every fix to be a real root cause.

It also captures a moment I'm proud of: rather than accept production changes that had been made to calm the tests, I proved empirically that most of them weren't needed and protected the production code from being changed to satisfy a test artifact. That is the kind of judgment I want recruiters to see.

Problem

The service's end-to-end suite was non-deterministic: a given run might produce anywhere from 25 to 37 failures, and a different set each time. Because the result was unreliable, CI couldn't be trusted. A red run might mean a real regression or nothing at all, so real regressions could hide in the noise. The suite exercised real dependencies (a real auth provider, a database, a local message broker, and an external sandbox API), any of which could contribute to the instability.

Constraints

—The failure was statistical, not reproducible on demand. You cannot fix what you can't reliably observe; the first job was to make the flakiness measurable.

—A flaky suite is a broken measurement instrument. Every "fix" is itself measured by the same unreliable suite, so I had to validate changes across many runs, not one.

—The easy fixes were all the wrong ones. Skipping the offending tests, relaxing assertions, or bumping timeouts would have turned the suite green while destroying its value. I ruled those out up front.

—Some noise came from real dependencies, so root causes ranged from test setup to production code to environment configuration.

Decision

I treated it as a diagnosis problem with a strict protocol.

—Reproduce and characterize statistically. Run the suite many times to measure which tests failed and how often, turning "it's flaky" into a ranked list of concrete offenders.

—Categorize by root cause, not by symptom. The causes turned out to be varied: a divergent database schema, an auth bypass returning success where it should have returned forbidden, invalid test credentials, missing seed data, hardcoded IDs, and an accidental field spread in a patch handler.

—Fix each cause at the source, incrementally, re-running to confirm each fix reduced failures without introducing new ones.

—Validate over multiple runs rather than one green run, since determinism is a property you demonstrate statistically.

—Documented the runtime environment variables the suite depended on, so its behavior stopped being folklore.

Trade-offs

—Root-cause fixes over masking (skip / weaken / inflate). Masking is minutes of work and destroys the suite's reason to exist; root-causing is slower but leaves a suite you can trust. I chose to constrain myself out of every shortcut.

—Empirically validating a peer's proposed production changes over accepting them. Testing each change cost time and a potentially awkward review conversation, but changing production code to satisfy a flaky test would have been the tail wagging the dog. I preferred to revert changes I could prove were unnecessary.

—Fixing the environment/setup over trusting the suite's assumptions. Documenting env vars and correcting seed/schema drift is unglamorous but removes whole categories of intermittent failure.

Impact

—302 tests passing, zero failures, deterministic behavior with the consumer running, from a suite that produced 25 to 37 varying failures per run. The 8 skips in the final count were pre-existing and left as they were; I added none.

—Backend CI became trustworthy again, so a red run once more means a real problem.

—Protected production code and business rules from being altered to appease a flaky test, by disproving three of four proposed changes in review.

—Qualitative: regressions are once again caught by the suite instead of hidden by it.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—A flaky suite is a measurement instrument you must first make trustworthy. Until it is, every result, including your own fixes, is unreliable. Characterize before you change.

—Fix the root cause; never weaken the test. Skips, relaxed asserts, and inflated timeouts convert a signal into silence.

—Never change production code to make a test pass until you've proven the code, not the test, is wrong, and you can prove it empirically.

—Determinism is demonstrated statistically. One green run proves nothing about a suite that was flaky; many do.

Evidence

—From 25 to 37 varying failures per run to 302 passed, 8 skipped, 0 failed with the consumer active, which was the acceptance criterion written into the ticket before the work and the number it closed on.

—Self-imposed constraints: no *new* skips, no weakened assertions, no inflated timeouts. The 8 skips were already there; I did not add to them or claim them away.

—Verified against the tracker (2026-06-02 to 2026-06-23): the recorded root causes are test setup/data plus one genuine race in a cancellation test (a 404-vs-202 flake where the operation left its pending state before the cancel ran). De-flaked by seeding the operation directly into its pending state through the repository, bypassing the message broker, which made the test deterministic instead of merely retried.

—Verified: three of four proposed production changes reverted as unnecessary: an auth-guard status remap, an ordering clause inside a subquery for the active cycle, and a filter in the location-options path, with the location-options tests rewritten to assert the behaviour that already existed. The one change kept was a lateral join resolving a version identifier in the cycle listing: reverting it would have reintroduced a known customer-reported bug, so it was kept, documented, and tracked in its own ticket rather than smuggled in with test fixes.

—Delivered as a single pull request in the backend service, the most-discussed of the epic (25 review comments).

—Source (private): Jira ticket in the inspection domain, 2026-06; consolidated career knowledge base.