Skip to content

FLT-182 baseline incident data (pre-tool)

Collected by WT-8f70 via peer-session survey, 2026-07-30. Seeds the rc incident retro backfill (§3.5). Counts are the measured "before" for evaluating the tool.

WT-64ed (WING-240 deploy session — the triggering afternoon; ~5h, one app, 4-5 sessions)

Own (4 incidents + 1 near-miss): 1. dirty-build — shared-checkout gcloud builds submit .; master moved under upload + peer's uncommitted site-content.json in tree. Caught by peer question only. 2. zombie-build — local kill left server build running; mislabeled tag pushed, deleted after. 3. other/serving-conflation — read latestReady 00034 as serving while traffic pinned to 00031; ~1h lost. 4. land-race/chat-wrongness — asserted stale facts (htmx 4.0, migration status) to 3 lanes as truth. 5. near-miss other/weak-verification — non-discriminating verification check; peer redirect saved it.

Witnessed (8): 6. mutable-tag-pin + swallowed exit code (WT-570b) — build && deploy :v2 | tail; failed build fell through, v2 resolved to prior day's image, prod silently rolled back ~24h. 7. other/traffic-race (WT-9265) — silent traffic pin to 00031 mid-incident, undisclosed to deploy holder. 8. env-key-loss (WT-9265, armed WING-247) — ladder declares 0/7 live reader keys; --set-env-vars replace would strip them; existing AssertReconcilable gate blind to keys. 9. land-race (WT-4ee3 vs WT-64ed) — same 4 files, order coordinated by chat; mid-flight ticket reassignment. 10. other/verification-staleness — bucket content rewritten under a "verified" claim (04:48Z vs 18:49Z). 11. other/duplicate-memory — 3 duplicate memory entries, one rule, one day. 12. mutable-tag-pin (standing) — prod service image ref = wwsite:v2 long before today. 13. deploy-adjacent — cloudbuild _TAG fail-closed worked, but pipeline continued past it (see 6).

Requirements extracted → spec §3.4b (S1–S6).

WT-9265 (MPA auth surfaces / WING-246 incident author)

Own: unilateral traffic pin (--to-revisions, would have made every later deploy serve 0%); 2 stale-verification claims; 2 land-order races; ~4 uncoordinated bucket rewrites; armed env-key-loss still live (0/7 reader keys declared); reverse mutable-tag hazard (publisher removed, stale tag lingers); 3 silent-green mechanisms (Stryker no-match, testhost-lock contamination 49.36% vs real 78.95%, agy argv-limit exit-0); 2 vacuous tests. Priority ask: provenance gate (refuse digest w/o in-project build record) + claim-gate traffic ops. Detail: FLT-182 comment + WING-246; continuation worktree wt-cont-wing243.

WT-8f26 (WING-219 env ladder) — 22 incidents, ~6 h, 0 outages

Correction to ticket premise: the 7-key strip was caught by a passing remark + hand diff, NOT red team (red team caught an earlier instance). Headline items: replace-all semantics weaponised pre-existing declaration gaps (6/19 and 0/7 keys); missing DatabaseId → qa would have hit prod Firestore; --ssl-certificates replace → prod TLS outage risk; url-map '*' fail-open to prod; hello-container promote near-miss (caught by unrelated drift check); sync-push "triggered ≠ pulled"; duplicate-deploy collision (3 consistent local views, 1 collision); misattribution structural (%an useless, rebase replays SHAs). Detector split: council 6, live-verification 5, peers 5, tooling 3, passing remark 1, coincidence 1 — only 6/22 via formal gate.

WT-8ccc (WING-214 OAuth lane)

No commit annotations on revisions (attribution = archaeology); 3 attribution errors 3 directions; --set-env-vars near-miss (23 vars incl. secret mount; used --update-env-vars + before/after diff); placeholder-promote near-miss; image-only deploy can't ADD required key (00027-sf2 start failure, traffic held); service Ready=True is not a rollout check (assert latestCreated==latestReady); native exit codes lie ($ErrorActionPreference ignores native exes); secret-version disable ≠ rollback (latest → FAILED_PRECONDITION); prod telemetry stamped environment=unknown → all prod metrics excluded prod (unknown duration); wrong diagnosis published to 2 tickets (derived-at-deploy config invisible in declarations); compose bind-mount landmine outside git. Top-3 ask: stamp provenance labels; verify by artifact post-deploy; refuse/confirm --set-env-vars, merge default.

WT-373f (WING-241 .seeded)

Runtime-write path inside tracked tree + broad git add -A → landed foreign runtime marker; single-branch fetch refspec → false-blocked lanes; stale merge-base rebases routine; claude -p backgrounded gates killed at exit reporting done; council seats die exit-0. Preflight asks: porcelain-clean on exact ref, immutable-SHA builds, runtime paths ignored, merge-base freshness + feature-only diff, refspec visibility.

WT-570b (WING-207 front-page/CMS lane)

Mutable-tag stale deploy (build failed, | tail swallowed exit, :v2 = day-old image, revision 00034 created; traffic pin saved it) + false "nothing changed live" report from output shape; revert-by-deploy near-miss (8-vs-6 divergence, deploy would have reverted live WING-240 — caught by manual merge-base check, nothing in path would warn); 2x peer uncommitted site-content.json in own worktree (wingcms MCP WINGCMS_CONTENT pointed at one worktree = accidental shared writer); stale repo copy published over bucket (shipped, user-visible → drove ETag redesign); ~15 CLI publishes with no concurrency check; divergent tool evolution found at rebase; CI gate gap (151+ tests never in ci.yml — all green claims CI-meaningless); CSP behavior flip from self-hosting htmx (infra change silently un-blocked admin hx-boost); 2-3x offload silent no-op exit-0. Lesson: every failure = one inference past a verified fact; make each fact a cheap query.

WT-ab3e (WING-227 MPA mobile lane)

Wrong-lineage branch (unrelated histories vs deployed lane; deploy would have reverted 3 live features — caught by peer pre-deploy); same-tag double-build race with WT-9265 (averted by direct deploy-lock message); unattributed revisions 00019→00023 (3-session investigation; shared gcloud identity → attribution by timing only); same commit landed twice as different SHAs (identical patch-id, benign by luck); stale board state nearly misrouted work; git add -A swept .seeded into deploy commit (amend+force post-push); 2x cwd drift ("not a git repository" masked a real ancestry check); CSS-probe verification blind spot (witnessed). Lesson: ancestry checks pre-build/deploy + per-service/per-tag deploy lock.

Aggregate baseline (pre-tool, ~2 days, one app family, 5 sessions reporting)

~70 events: 13 (WT-64ed) + 22 (WT-8f26) + ~15 (WT-8ccc) + ~6 (WT-9265 distinct) + ~5 (WT-373f) + ~10 (WT-570b). Reached production/users: 3 (stale-tag revision created — traffic pin prevented serving; telemetry environment=unknown; stale repo copy over bucket reverting a live fix). Armed at survey time: 1 (env-ladder 0/7 reader keys, WING-247). 2 MPA-lane surveys still queued; fold in on reply.