Making container and dependency hardening a standing practice, not an audit response

claim 1.8dependent — cites claim 1 (Dynamox)

Turned container and dependency hardening into a standing quarterly practice across two backend services over four quarters, covering non-root runtimes, 135 CVEs flagged and all 7 rated critical remediated in the first pass, then a base-image and dependency overhaul that took the audit from 69 findings to 16, unfixable OS CVEs from 160 to 0, and the final image from 1.64 GB to 463 MB, with exploitability-based accepted risk documented rather than version numbers chased, and one self-inflicted rollout regression root-caused and fixed the same day.

impactFirst pass (2025): 135 CVEs flagged across two services, all 7 rated critical remediated; runtimes switched to a non-root user; base image modernized.Latest pass (2026-08): dependency audit 69 → 16 findings; OS-level CVEs with no upstream fix 160 → 0; final image 1.64 GB → 463 MB (−72%); four criticals removed outright by purging one unused base package.Six advisories consciously left open with written, exploitability-based justification, which is the part of the work that keeps the report trustworthy.Security hardening became a standing part of how backend services ship, across four consecutive quarterly "vulnerability reduction" epics from 2025Q3 onward, one of the six areas cited in my promotion to mid-level, and by the last pass automated with a purpose-built AI skill.One self-inflicted staging outage, root-caused and fixed the same day; nothing reached production.

Context

This is my evidence for security as an engineering practice rather than a fire drill. Security was one of the six competency areas in my promotion dossier, and this is the concrete work behind it, but the part I'd actually defend in an interview is what happened *after* the promotion: the same practice ran for four more quarters, and the last pass is where the judgment shows. Not every advisory deserves an upgrade, not every CVE is reachable in your code, and a base image that eliminates 160 unfixable vulnerabilities can still take your service down if you don't check what your entrypoint depends on.

Problem

Backend services ran in containers on Kubernetes without a formal hardening or CVE-remediation practice. Vulnerability exposure in base images and dependencies is the easiest kind of debt to defer: it doesn't fail a build, a test, or a deploy, it just accumulates until an audit or an incident forces attention.

Concretely, at the start: images ran as root by default, the container-scan output had never been triaged, and no one owned the recurring work.

Constraints

—The exposure was spread across two services and their full dependency trees, not a single fixable line: 135 CVEs flagged in the first scan, of varying severity, and 69 audit findings still open a year later once the toolchain had moved on.

—Non-root containerization touches the whole runtime, not just the Dockerfile: file permissions, process ownership, and anything that assumed root has to be re-verified.

—Remediating a CVE often isn't a version bump. Some advisories have no fix in the major version you're pinned to; some are fixed only in a framework line you can't move to; some sit in operating-system packages the distribution has never patched. Chasing the number is not the same as reducing risk.

—Prioritization mattered. With 135 flagged, treating all of them as equally urgent would have meant treating none of them as urgent.

—A base image is a runtime contract. Swapping the distribution changes which binaries exist, and the things that break are not in your code, they're in the deploy manifest.

Decision

2025, establish the floor. I mapped the container-scanner output for both services into tracked work rather than leaving it as a report, switched the runtime to a non-root user, moved to a slimmer base, and worked the critical findings first: a cluster of Kerberos library CVEs, a Berkeley DB CVE, a form-data advisory in both services, a test runner advisory. Seven critical, all closed. Then I turned it into a recurring quarterly epic ("vulnerability reduction") rather than a finished task, which is the only reason the practice survived past the audit that prompted it.

2026, attack the causes, not the count. By the last pass the interesting findings weren't application dependencies at all:

—Transitive dependencies got pinned, not upgraded. Package-manager overrides fixed the reachable production dependencies without a breaking change, and separately pinned a batch of toolchain packages that never reach the runtime.

—The operating system was the real problem. The Debian-slim base carried 160 OS-level CVEs with no upstream fix available, unactionable by definition. Moving to an Alpine base took that to zero, because every remaining advisory had a fix. I also purged a base package that carried four criticals Debian had never patched.

—The image was carrying its own build. Docker layers are cumulative, so devDependencies and a recursive ownership change stayed in the image even after being pruned. Splitting the Dockerfile into build and runtime stages took the final image from 1.64 GB to 463 MB.

—I wrote down what I was not fixing, and why. Four toolchain packages had no fix in the pinned major and an upgrade would have been disproportionate. Two framework packages had fixes only in a major line the service can't move to, and the vulnerable code path is the SSE/TCP transport, while this service only ever uses the Kafka transport, so the path is never exercised. That reasoning is in the pull request, as accepted risk with a stated justification, not omitted to make a number look better.

Trade-offs

—Alpine over Debian-slim, chosen because it moved 160 *unfixable* OS CVEs to zero, which is a real risk reduction rather than a lower number. Accepted cost: a different libc, which meant verifying the database client's native engine and, as it turned out, a missing shell in the deploy path.

—Pinning transitive dependencies via overrides over upgrading the direct dependents, giving no breaking changes and immediate coverage, at the cost of a lockfile that now encodes decisions someone has to revisit.

—Documented accepted risk over chasing a clean report. Leaving six advisories open with written justification (no fix in the pinned major; vulnerable transport never used) is more honest and more useful than an upgrade that destabilizes the service to make a dashboard green.

—Non-root images over root (the default), for a smaller blast radius if a container is compromised, at the cost of re-verifying everything that assumed root.

—Remediating by severity over volume, taking the 7 criticals first even though it left lower-severity items open longer.

—A recurring quarterly epic over a one-time cleanup. More process overhead, but it's the difference between a practice and an anecdote. Four quarters later it's still running, most recently executed with an AI skill built for the purpose.

Impact

—First pass (2025): 135 CVEs flagged across two services, all 7 rated critical remediated; runtimes switched to a non-root user; base image modernized.

—Latest pass (2026-08): dependency audit 69 → 16 findings; OS-level CVEs with no upstream fix 160 → 0; final image 1.64 GB → 463 MB (−72%); four criticals removed outright by purging one unused base package.

—Six advisories consciously left open with written, exploitability-based justification, which is the part of the work that keeps the report trustworthy.

—Security hardening became a standing part of how backend services ship, across four consecutive quarterly "vulnerability reduction" epics from 2025Q3 onward, one of the six areas cited in my promotion to mid-level, and by the last pass automated with a purpose-built AI skill.

—One self-inflicted staging outage, root-caused and fixed the same day; nothing reached production.

Lessons Learned

Reusable engineering knowledge I carry forward from this:

—Vulnerability exposure accumulates silently. Nothing in a normal build/test/deploy cycle forces attention to it, so it needs its own standing practice with a recurring slot rather than a response to an audit.

—A CVE with no upstream fix is not a task, it's a base-image decision. 160 unactionable advisories aren't a backlog to work through; they're a signal that the distribution is the problem.

—Argue exploitability, not version numbers. "Fixed only in the next major, and the vulnerable transport is never used in this service" is a stronger position than a disruptive upgrade, but only if you write it down where a reviewer can challenge it.

—Docker layers are cumulative, so pruning inside one layer prunes nothing. Multi-stage is how you actually remove build-time weight from what you ship.

—A base image is a runtime contract. What breaks when you swap it lives in the deploy manifest, not in your code, and it only breaks when a rollout replaces the pods, which is the worst possible moment to find out.

—Non-root by default is a runtime decision, not a Dockerfile line. It has to be verified across the whole service, not just declared.

Evidence

—2025 pass: container-scanner findings mapped into tracked work for both services; non-root runtime; the 7 critical CVEs closed as individually tracked items (Kerberos library cluster, Berkeley DB, form-data in both services, test runner), all with merged pull requests.

—2026-08 pass, verified from the pull request itself: audit 69 → 16; OS CVEs with no fix 160 → 0; image 1.64 GB → 463 MB; before/after scanner output attached; validated with a full build and lint, 1,870 unit tests at zero failures, the e2e suite at 328/330, a real container build, and a live API boot inside the new image.

—Accepted risk documented in the pull request, not omitted: six advisories with written justification, including the reachability argument for the two framework packages.

—Regression owned: the Alpine swap crash-looped three staging deployments because the Kubernetes entrypoints invoke `bash`; root-caused from the pod events and fixed in a follow-up the same day, with the concurrently released change explicitly ruled out.

—Practice, not project: four consecutive quarterly "vulnerability reduction" epics from 2025Q3 onward, the most recent (2026-08) executed by running a purpose-built AI skill against the service.

—Cited as one of six competency areas (security) in my promotion dossier (see companies/dynamox.md).

—Source (private): Jira epics and tasks in the inspection domain, 2025-07 to 2026-08; the corresponding pull requests in both backend service repositories; consolidated career knowledge base.