Skip to content

Release pipeline (canonical deploy pattern)

WorkWingman cloud edition runs on GCP Cloud Run behind Cloudflare. This document is the engineer-facing deploy contract, frozen from the 2026-07-30 incident review (WING-240/246). Do not invent steps past these facts.

Canonical deploy pattern

  1. Build from a pinned clean worktree at an exact commit. Porcelain-clean only. Never build from a shared checkout — a peer's uncommitted content nearly shipped once.
  2. Image tag is sha-<commit> via _TAG substitution. Mutable tags are banned. A v2 tag re-fired old bytes and was deleted.
  3. Deploy by image digest, not tag.
  4. Env-diff before traffic. Compare declared env vs live service config. The env-ladder once armed stripped 7 live reader settings (RT-3 hazard).
  5. Allocate new revision at 0% traffic, verify, then shift traffic explicitly. serving revision and latest-ready are distinct states — check both.
  6. Verify by a content discriminator (e.g. Jira href), not HTTP 200 alone.
  7. Exit-code discipline. Deploy scripts must not swallow failures. Silent rollback root cause was exit-code swallowing.
  8. Traffic pins silence all subsequent deploys. update-traffic to a fixed revision leaves new revisions at 0%. After break-glass, must --to-latest. Break-glass ops are announced, never silent.

Seven-mode incident taxonomy (2026-07-30, one session)

# Mode What happened
1 Unattributed revisions Mystery revision 00034; prod served ~24h-stale image
2 Mutable-tag time bombs Stale-byte dispensers with no active publisher
3 Traffic ops > deploys Pins silencing future deploys
4 Config drift invisible until deploy Undeclared live env keys
5 Land-order races Peer merge shipped a version with an already-fixed bug
6 Vacuous tests Test passes by doing nothing on invalid seed; green while measuring nothing
7 Verification staleness Claims invalidated when code commit, image digest, or content generation move independently — track all three

Follow-on controls (FLT-182)

Per-app release coordination tool. Top two controls:

  1. Provenance gate — build/deploy only from known commit + clean tree + digest.
  2. Traffic claim-gating — no silent pin; no unowned traffic state.

Make facts cheap so nobody infers one step beyond facts.

Non-negotiables (checklist)

  • [ ] Clean pinned worktree at exact commit
  • [ ] Tag = sha-<commit> only (no mutable tags)
  • [ ] Deploy digest, not tag
  • [ ] Env-diff declared vs live before traffic
  • [ ] New revision at 0% → verify → explicit traffic shift
  • [ ] Check both serving revision and latest-ready
  • [ ] Content discriminator verification
  • [ ] Non-zero exit on failure (no swallowed exits)
  • [ ] No silent traffic pins; announce break-glass; --to-latest after pin