Shipping AI Code Safely: Six-Month Infrastructure Review

Shipping AI Code Safely: Six-Month Infrastructure Review (Jan–Jul 2026)

Author: Bomee Jung · rev. 2026-07-05

Six months ago we committed to AI-assisted development as the default way we build Momentum. The bet was never that AI code is safe on its own — it isn’t — but that a team can build enough scaffolding around it that speed and quality stop being a trade-off. This memo reviews what got built, when, and what visibly changed as a result.

The companion doc AI Code Generation Practices covers the day-to-day practices; this memo is the retrospective on the infrastructure and its measured outcomes.

TL;DR

• In Q1, AI-assisted velocity outran our safety net: the bug/PR ratio quadrupled (0.07 → 0.26), February alone produced 143 bug tickets, and 43% of bugs were being caught by QA/PM on staging rather than in review or CI.
• We responded with layered infrastructure: an AI-legible codebase (docs, pattern library, skills), machine-enforced conventions (arch tests with a ratchet, behavioral-test gates, two-pass AI review), earlier detection (restructured CI, per-merge browser suite, Sentry gates), spec/test honesty tooling (OpenSpec coverage checkers), and process automation (daily grooming, release pipeline).
• The outcomes are visible: open P0/P1 bugs went from holding at 26–28 in March to 0–4 from May onward; regression bugs fell from 16/month to 7/month; true cross-release regressions are down to ~2 per release; and releases went from all-hands events to “9 releases in 14 days” in May, now effectively daily — while team output roughly doubled.
• Not everything worked on the first try. We rebuilt the docs MCP from scratch, retired /stage-release, downgraded a too-brittle CI gate, and added adversarial verification because agents hallucinate. Those corrections are part of the system now.

Where we started: the January–February problem

The velocity gains showed up before the safety net did.

• Bug volume 6x’d quarter over quarter: ~49 bug tickets in Q4 2025 → 289 in Q1 2026, peaking at 143 in February (team-pulse reports/quarterly/momentum/2026-Q1-bug-trends.md).
• Bug/PR ratio quadrupled from 0.07 (Oct 2025) to 0.26 (Mar 2026).
• Bug creation more than doubled in February (36 → 86) even as we merged 385 PRs from 6 developers.
• Detection was happening too late: 43% of bugs were found by QA and PM on staging. We framed this internally as a detection-timing problem, not a bug-rate problem — the code review layer was passing things that only surfaced two stages later.
• AI-authored PRs needed real scrutiny: a March audit of Copilot-authored PRs found issues in every one reviewed — 5 with incomplete implementations, 4 with broken tests, 2+ with architecture violations, 3 gone stale.

The Q1 lookback named the tension plainly: we were shipping faster than our infrastructure could support. Everything below is the response.

What we built, layer by layer

1. An AI-legible codebase

The premise: AI doesn’t know our codebase, and the fix isn’t better prompting — it’s better documentation, structured so agents consume it.

• momentum-docs MCP + design-system docs (Feb) — a searchable HOW-guide and router covering 229 components across 27 categories, so AI queries conventions instead of guessing. Rebuilt from scratch in March after we measured accuracy at 40–95% and reoriented it to HOW-questions with OpenSpec as the WHAT.
• Code-quality pattern library (momentum/.claude/data/code-quality-log.tsv) — every reviewed AI mistake worth remembering becomes a logged pattern (24 entries: logic, design-system, test, PHPStan, perf, a11y; 15 major). The review-ai-pr skill loads this as a checklist on every AI-authored PR review.
• CLAUDE.md as process-as-code — conventions that used to live in people’s heads (PR typing by UAT surface, DDD placement rules, notification patterns, “no identifier fallback chains”) are written where agents read them. Pruned from 53.8k to 38k characters in June so the signal survives.
• ~24 skills in momentum alone encoding whole workflows: make-ui, make-admin, make-seeder, ddd-review, find-n-plus-one, land-pr, create-feature, root-cause, and more.

The operating rule throughout: fix the class, not the instance. When AI (or a human) gets something wrong, the correction becomes a rule, a skill, a lint check, or an arch test — so that mistake class can’t recur silently.

2. Machine-enforced conventions

Documentation is advisory; these are not.

• Architecture tests (tests/Arch.php, 114 test blocks) — DDD bounded-context isolation, framework-free domain layers, service-layer write paths, notification patterns, schema-only migrations, banned footguns (CreateNewUser for bulk creation, per-feature notifier ports). Several were written in direct response to observed AI failure modes.
• The exemption ratchet (tests/.arch-exemption-baseline) — legacy exemptions are counted and the count can only go down; raising it requires editing the baseline file in the same PR, in plain view of review. Currently at 123. During the May–June refactor wave the ratchet dropped 42% in two weeks — the tests visibly doing their job.
• Behavioral-test gate (require-bug-fix-test.yml, Mar) — bug-fix PRs must include a test that fails without the fix.
• Two-pass AI review on every PR (Mar) — two independent models review each PR before a human sees it.
• Risk-based review tiers (PR template): zero-risk (AI review or none) → low (AI review) → medium (AI review + 20% post-merge sampling, enforced by a sampler workflow) → high (named human reviewer with a specific question). This is how we spend scarce human review attention where it matters.
• Allowlist hygiene — every arch-test allowlist addition requires a removal ticket or an explicit permanent-exemption rationale; silent erosion is not an option.
• Guardrails beyond code shape: user-facing error catalog with PHPStan leak-guard rules (no raw getMessage() to users), a warn-only questionnaire PII guard, feature-flag manifest with a CI code-gate and lifecycle policy, CSP flipped from report-only to enforced (June 30), and CodeQL on a weekly schedule.

3. Earlier detection

The answer to the “43% found on staging” finding was to restructure when tests run, not just add more.

• CI test-strategy OpenSpec (#12557, Mar) — fewer per-PR tests, with a comprehensive suite running on a schedule against main. Trimmed from 8x to 3x daily in July once the per-merge layers below made the cron partly redundant.
• Smoke gate on every PR + flake telemetry (Testing CI Phase 0, Jul) — fast confidence per PR; retried-pass tests are recorded, not failed, so flakes become data instead of noise.
• Per-merge browser suite on main — full Playwright run on every push to main, giving per-merge blame within minutes instead of the ~8 hours the scheduled suite implied.
• Sharded, change-aware CI — 8 feature-test shards in full mode, collapsing to 1 for docs/script-only PRs; integration tests sharded 4-way with changed-file gating.
• Automated UAT E2E suite (June) — seeded, cron-scheduled browser flows for admin/auth, collections, buildings; layer-3 role-based authorization flows wired into CI.
• /main-sentry-check as a release gate (June) — before promoting, verify every Sentry error that appeared on main since the last staging release has a tracked issue. Added to the release checklist as a pre-promote gate.

4. Keeping specs and tests honest

AI writes plausible tests; plausible is not the same as verifying. OpenSpec (136 published specs, ~150 archived changes) is the WHAT, and a tooling layer keeps it tied to reality:

• @scenario tags + scenario-coverage.py — tests declare which spec scenario they verify; the checker buckets every scenario as Verified / Heuristic / Gap / Dangling.
• spec-drift / main-spec-drift — flags spec scenarios with no matching tests, and drift between main and published specs.
• verify-coverage — goes a step further than keyword matching: dispatches an agent to read each test body and judge whether its arrange/act/assert actually exercises the scenario.
• /spec-audit — a scheduled audit producing a digest of archival candidates and code/spec drift; the June 4 run audited 25 changes, applied decisions to 14 specs, and deleted dead code the audit surfaced.
• Spec PRs ship separately from code PRs (daisy-chained) so design review and code review get different reviewers and different scrutiny — an AI-authored 70-file diff can’t hide under a “docs” title.

5. Automating the process around the code

Human judgment is the scarce resource; the automation’s job is to spend it only where judgment is actually required.

• Daily grooming Action (June, team-pulse) — four audited, apply-gated engines: close issues whose no-UAT-type PRs merged; enforce PR↔ticket link hygiene (SOC 2); auto-create UAT testing tickets from merged feat/perf PRs’ Testing sections; fix forward board-status drift. Judgment-shaped cases are staged into a weekly digest instead of auto-acted. The routing runs on PR type prefixes — which is why feat: vs chore: typing now matters operationally.
• Release pipeline redesign (July 3, momentum #18481 + team-pulse #44) — deterministic cut with cycle artifacts (QA checklist, flag checklist, notes draft), hard prod gates (staging-at-tag, suite green, flag decisions made, zero milestone Release Blockers), and a re-cut engine for QA-window patches (cherry-pick, bump patch tag, atomic push — never force).
• Cut-watch + post-release routines — a daily job computes release signals (batch size/type/risk, tag age, blockers, suite status) and proposes a cut on the release checklist; after a prod release, housekeeping runs and a draft regression report is posted for the retro. Design intent: humans never originate — every decision arrives as a prompt. Exactly two human gates remain: cut approve/defer, and flag go-live.
• Backlog and Sentry hygiene — /sentry-triage, /triage, and periodic amnesty campaigns (597 stale issues closed in one day in May; the open count is now ~388 against 9,400+ closed).

6. Measuring whether it’s working

• team-pulse — a single SQLite warehouse of issues, PRs, commits, board state, and code complexity, with extractors, a velocity dashboard, and monthly/quarterly quality reports. The bug-trend numbers in this memo come from it.
• Vibemaxx (Apr) — daily scorecards during the vibe-coding push: rebound rate (prod bugs per release), regression tracing via git blame, per-person AI adoption, bottleneck ratio.
• Origami — per-person capability self-assessments, so we can see whether the team is learning the system, not just using it.

Milestones with observed quality impact

The sequence matters: measurement came first, then enforcement, then detection timing, then process automation. Each layer’s impact is visible in the numbers the previous layer let us see.

When	Milestone	Observed outcome
Jan	SOC 2 agentic-coding policy; issue taxonomy; team-pulse metrics DB	Baseline established — this is how we know the Feb numbers at all
Feb	AI-legibility push (docs MCP, pattern rules); Filament removal + first arch-test wave	Bug creation peaked (86 created in Feb); the class-not-instance loop started
Mar	CI test strategy #12557; behavioral-test gate; two-pass AI review; adversarial verification	Regression bugs fell 16/mo → 7/mo (18% → 10% of bugs); March bugs down 70% vs Feb (82 → 25); open P0/P1 began falling from the 26–28 plateau
Apr	Vibemaxx week — the deliberate throughput experiment with the safety net on	Team output +103% PRs (excluding me), AI adopters 1 → 5, issue burn-down 2.8x baseline — with rebound tracked release-by-release; true cross-release regressions measured at ~2 per release
May	Release muscle + refactor wave under arch-test protection	“9 releases over 14 days — finally a no-big-deal push-button exercise”; then daily. Arch-exemption ratchet down 42% in the May 23–Jun 6 window. Open P0/P1 hit 0 the week of May 21
Jun	Grooming Action; flag CI gate; UAT E2E suite; Sentry release gate	Board/ticket state now maintained by machine (misrouted UAT items — e.g. a seeder PR tagged perf: clogging the UAT queue — became a solved class); open P0/P1 holding at 1–4
Jul	Release pipeline redesign; cut-watch/post-release; CI cost tuning (8x → 3x cron)	Release process runs on two human decisions; regression retro input is auto-drafted; /stage-release retired

The headline trend: open P0/P1 bugs went 26 → 28 → 25 → 27 through March, then 3, 2, 7, 6, 2, 0, 1, 4, 3, 2 from April through late June (weekly updates), while release cadence increased roughly 10x and team PR output roughly doubled. Q1’s bug-resolution-time debt (median close time rose to 88 days by March) is the cost we paid for the February spike; the near-zero open-P0/P1 state since May is what paying it off looks like.

What we changed course on

Listing these because the corrections are load-bearing — the system works because it gets revised when it’s wrong:

• momentum-docs MCP was rebuilt from scratch after accuracy measurement (40–95%) showed the first version couldn’t be trusted as a router.
• /stage-release was retired (July) — first its CI workflow was deprecated (June 14), then the whole command was absorbed into the automated cut path. Its team-pulse audit fork was deleted in favor of a single source of truth.
• The @scenario drift check was downgraded from a hard CI failure to a coverage-report surface — the strict gate was too brittle to live on every PR.
• claude-review was gated to manual-trigger only immediately after installing the GitHub workflow; always-on wasn’t the right default.
• Adversarial verification exists because agents hallucinate: the first regression-report agent produced fabricated commit hashes and a wrong architectural framing. Antagonistic-verification agents now fact-check the analysis agents, and they catch things.
• Grooming needed several rounds of hardening — diagnostics-only PRs wrongly flipping tickets to Ready to Test, task-list checkboxes wrongly holding issues open, a board-wipe-on-fetch-failure bug. Each fix is now permanent behavior.
• The measures refactor failed four times before it succeeded — the fifth attempt worked because it started with guardrail tests and a fresh spec rather than more effort.

What’s still open

• Bus factor on the meta layer. The CI gates, arch tests, review workflows, and grooming/release Actions are team infrastructure that runs on its own. But the measurement layer (vibemaxx, origami, spec-drift tooling, the team-pulse reports) has effectively one maintainer. The H2 goal is that anyone can run a release and the stack survives a two-week absence — the July pipeline redesign is the biggest step toward that so far.
• Monitoring and on-call didn’t keep pace. PagerDuty has been a recurring open action item since January; Sentry alerting has had silent gaps. The Sentry gates added in June/July help, but production observability is still behind the rest of the system.
• Complexity hotspots grew while quality debt was paid down elsewhere: functions with cyclomatic complexity >10 tripled during 2025 (57 → 205). The ratchets cover architecture boundaries, not yet hotspot growth.
• Grooming automation covers momentum only — momentum-calcs still relies on local runs pending the GitHub App install.
• Q1’s slow-close backlog (P3 median close time reached 187 days) is still being worked down through triage and amnesty cycles.

The operating principles, for the record

1. Fix the class, not the instance. Every escaped defect should end its life as a rule, gate, skill, or test.
2. Detection timing beats detection volume. Move the catch earlier: review → PR CI → per-merge → scheduled → staging → prod, in that order of preference.
3. Ratchet, don’t aspire. Counted baselines that can only decrease (arch exemptions, PHPStan suppressions — down ~92% in H2 2025) outperform coding-standards documents.
4. Spend human judgment only where it’s required. Deterministic detection + safe action → automate fully. Deterministic detection + judgment-required action → stage it for a human. Anything else stays human. A wrong auto-close erodes trust in the entire system.
5. Verify the verifier. AI review, AI reports, and AI tests all need an adversarial layer — agents fact-checking agents is now standard for anything that feeds a decision.
6. Measure before and after. None of the claims in this memo would exist without the metrics warehouse having been built first.