Release pipeline (canonical deploy pattern)¶
WorkWingman cloud edition runs on GCP Cloud Run behind Cloudflare. This document is the engineer-facing deploy contract, frozen from the 2026-07-30 incident review (WING-240/246). Do not invent steps past these facts.
Canonical deploy pattern¶
- Build from a pinned clean worktree at an exact commit. Porcelain-clean only. Never build from a shared checkout — a peer's uncommitted content nearly shipped once.
- Image tag is
sha-<commit>via_TAGsubstitution. Mutable tags are banned. Av2tag re-fired old bytes and was deleted. - Deploy by image digest, not tag.
- Env-diff before traffic. Compare declared env vs live service config. The env-ladder once armed stripped 7 live reader settings (RT-3 hazard).
- Allocate new revision at 0% traffic, verify, then shift traffic explicitly.
serving revisionandlatest-readyare distinct states — check both. - Verify by a content discriminator (e.g. Jira href), not HTTP 200 alone.
- Exit-code discipline. Deploy scripts must not swallow failures. Silent rollback root cause was exit-code swallowing.
- Traffic pins silence all subsequent deploys.
update-trafficto a fixed revision leaves new revisions at 0%. After break-glass, must--to-latest. Break-glass ops are announced, never silent.
Seven-mode incident taxonomy (2026-07-30, one session)¶
| # | Mode | What happened |
|---|---|---|
| 1 | Unattributed revisions | Mystery revision 00034; prod served ~24h-stale image |
| 2 | Mutable-tag time bombs | Stale-byte dispensers with no active publisher |
| 3 | Traffic ops > deploys | Pins silencing future deploys |
| 4 | Config drift invisible until deploy | Undeclared live env keys |
| 5 | Land-order races | Peer merge shipped a version with an already-fixed bug |
| 6 | Vacuous tests | Test passes by doing nothing on invalid seed; green while measuring nothing |
| 7 | Verification staleness | Claims invalidated when code commit, image digest, or content generation move independently — track all three |
Follow-on controls (FLT-182)¶
Per-app release coordination tool. Top two controls:
- Provenance gate — build/deploy only from known commit + clean tree + digest.
- Traffic claim-gating — no silent pin; no unowned traffic state.
Make facts cheap so nobody infers one step beyond facts.
Non-negotiables (checklist)¶
- [ ] Clean pinned worktree at exact commit
- [ ] Tag =
sha-<commit>only (no mutable tags) - [ ] Deploy digest, not tag
- [ ] Env-diff declared vs live before traffic
- [ ] New revision at 0% → verify → explicit traffic shift
- [ ] Check both serving revision and latest-ready
- [ ] Content discriminator verification
- [ ] Non-zero exit on failure (no swallowed exits)
- [ ] No silent traffic pins; announce break-glass;
--to-latestafter pin