Release reliability — cost of incidents, value of controls¶
Audience: business / founder. Scope: WorkWingman cloud edition (GCP Cloud Run behind Cloudflare). Facts from the 2026-07-30 deploy and incident review (WING-240/246; taxonomy from one session). No invented metrics beyond what is stated here.
What went wrong (near-miss and real lag)¶
| Failure mode | Business meaning | Observed impact (facts) |
|---|---|---|
| Unattributed revisions | Nobody can answer “what is live?” | Mystery revision 00034; production served ~24h-stale image |
| Mutable tags | Old code can reappear under a familiar name | Stale-byte re-fire via banned v2-style tag pattern |
| Traffic pins | Future releases look “done” but serve nothing | Pins silence all subsequent deploys (new work at 0%) |
| Config drift | Live behavior ≠ declared product config | Undeclared live env keys; env-ladder once stripped 7 live reader settings |
| Land-order races | “Fixed” bugs reappear after a peer merge | Already-fixed bug shipped again via merge order |
| Vacuous tests | Green CI that measures nothing | Pass-by-doing-nothing on invalid seed |
| Verification staleness | “Verified” claims expire when pieces drift | Commit, image digest, and content can move independently |
Headline reliability cost already paid: ~24 hours of production serving a stale image, plus near-miss of shipping another peer’s uncommitted work, plus silent-rollback risk from scripts that swallowed failures.
What the controls buy¶
| Control | Prevents | Business return |
|---|---|---|
| Build only from pinned clean worktree at exact commit | Unowned / dirty-tree deploys | Stops “who changed prod?” and peer-leak near-misses |
Immutable sha-<commit> tags; deploy by digest |
Mutable-tag time bombs | Stops old bytes resurfacing under a friendly label |
| Env-diff before traffic | Silent config strip / undeclared live keys | Protects reader settings and declared product behavior |
| 0% revision → verify → explicit traffic; check serving vs latest-ready | False “shipped” states | Ensures customers actually get the intended revision |
| Content discriminator (not HTTP 200 alone) | Vacuous “site is up” checks | Confirms the right product content is live |
| Exit-code discipline (no swallowed failures) | Silent rollback | Failures surface; rollback is a decision, not an accident |
Announce break-glass; --to-latest after pins |
Silent traffic locks | Restores normal release flow; no quiet freeze of all deploys |
Portfolio view (why this is not “ops trivia”)¶
- Stale production is unpaid debt: customers and demos see yesterday’s product while the board thinks today shipped.
- Unowned revisions and silent pins burn trust between people and between systems: status reports become fiction.
- Green-but-empty tests and HTTP-only checks create false confidence — the expensive kind, because it delays discovery.
- Config drift turns a “successful deploy” into a product regression without a code change.
Follow-on investment (FLT-182)¶
Per-app release coordination tool. Top two controls by leverage:
- Provenance gate — only builds with known origin ship.
- Traffic claim-gating — traffic ownership is explicit; pins cannot silence the pipeline quietly.
Design principle: make facts cheap so operators and agents never need to invent one step past evidence.
What leadership should expect after controls land¶
- A clear answer to “what commit, digest, and content are live?”
- No silent production freezes after emergency traffic moves
- Deploy failures that fail loudly
- Fewer “we thought it shipped” postmortems of the seven modes above
Anything not listed here as measured impact is TODO(verify) — do not treat this as a full financial loss model until instrumented.