Skip to content

FLT-182 — Release Coordinator (release-coordinator)

Status: DRAFT spec, pre-kerr-lens, pre-council. Author: Clahadore (WT-8f70), 2026-07-30. Ticket: FLT-182

1. Problem

One afternoon of WorkWingman deploys produced six coordination failures (ticket §"What happened today"): dirty shared-checkout build, zombie server-side build, unattributed revisions, mutable-tag prod pinning, env-var replacement stripping live keys, and sessions racing land order by chat.

Root cause class: many autonomous sessions share one app's release path (merge → rebase → ff-push → build → deploy) with no per-app serialization, no provenance enforcement, and no attribution.

1b. Mission (Andrew, 2026-07-30)

rc is the end-all-be-all for managing all fleet releases. Every mutation that changes what users or systems are served — code deploys, image builds/tags, traffic splits, bucket/content publishes, config/secrets, and in time DNS/CDN — flows through rc's claim + gate + provenance + ledger pipeline, on every app, on every PC. Architectural consequence: surfaces are pluggable — each release surface implements the same contract (claimable intent, preflight gates, execute-by-verified-artifact, provenance record, ledger wiring) so adding a surface never bypasses the pipeline. Anything not yet gated is listed in rc audit as known-open, never silently out of scope.

2. Decision: standalone repo

~/source/repos/release-coordinator, C# .NET 10, single-file exe release-coordinator.exe (alias verb rc), installed to C:\Tools\release-coordinator\ via install.ps1, distributed via OneDrive pool — identical skeleton to worktree-identity. NOT a fleet-harness module: fleet-harness is Python/LangGraph; mixing a C# csproj into it buys nothing and complicates CI. Integrates with worktree-identity (reads its registry for WT-id/ticket), fleet-repo-sync (push fan-out unchanged), council gate (deploy is a gated op).

3. Components

3.1 Per-app claim/queue (rc claim / rc release / rc queue)

  • App registry: apps.json (checked into release-coordinator repo, git-replicated): app name → { repo path, deploy platform (cloudrun), service name, project, region, protected branches }.
  • Claim store: git-backed, in a dedicated coordination repo path (release-coordinator repo, state/claims/<app>.json), because claims must be visible fleet-wide and git push/pull is the only cross-PC channel already proven (fleet-repo-sync). A claim = commit+push of the claim file; conflict on push = you lost the race, pull and queue. This makes git the arbiter — no new network service, no new trust surface.
  • Claim record: { app, wtId, ticket, intent: merge|rebase|ff-push|build|deploy, host, acquiredUtc, ttlMin (default 60), queue: [waiting claims] }.
  • rc claim <app> --intent deploy → if free: write+push claim. If held: append self to queue array, push, print holder + position. rc release <app> → pop queue head (if any) becomes holder.
  • TTL expiry: expired holder may be evicted by next claimant (rc claim auto-evicts stale + logs it). Heartbeat: rc heartbeat <app> extends; hook can renew like worktree-identity leases.
  • Enforcement (default-deny, Kerr fix K1): PreToolUse hook (Bash|PowerShell matcher) DENIES the deploy-verb classes everywhere, regardless of repo registration: gcloud run deploy, gcloud builds submit, gcloud run services update, docker push to known prod registries, and git push to a protected branch of a registered app repo. The only allow path is invocation via rc itself: rc exports a per-invocation token (RC_GATE_TOKEN, random, written to a session-scoped file the hook verifies and rc consumes) when it shells out to gcloud. Rewording the command or using an unregistered checkout no longer bypasses the gate — the verb class is the trigger, not the repo. JSON-deny on stdout, exit 0, same transport as worktree-identity PreToolUseGuard.
  • Anti-camping (Kerr fix K2): claims are intent-scoped and short — default TTL 15 min, heartbeat-renewed automatically only while rc commands for that app are actually executing (rc renews on each gated op). Idle holders expire fast; "claim the app for the afternoon" costs continuous activity, not one command.

3.2 Build provenance gate (rc build <app>)

Wraps the artifact build (gcloud builds submit / docker build): 1. Refuse unless cwd is a worktree registered to this session (worktree-identity registry lookup), git status --porcelain empty, HEAD attached, on a named commit. 2. Tag is DERIVED, never passed: <app>:<shortsha> from git rev-parse HEAD at submit time; also re-verify HEAD unchanged after upload manifest is computed (kill "tree moved under upload" class). 3. Emit provenance record state/builds/<app>/<sha>.json: { sha, wtId, ticket, imageDigest, utc }. Digest captured from build output; build without captured digest = failed build. 4. Bare gcloud builds submit . in registered app repos: PreToolUse hook denies, points at rc build.

3.3 Deploy-by-digest (rc deploy <app>)

  1. Deploy argument is a digest (@sha256:...), resolved from a provenance record. Deploying by mutable tag refused. rc deploy <app> --sha <shortsha> looks up digest from 3.2 record.
  2. Mutable-tag detector: rc audit <app> inspects serving revision's image ref; alerts if tag-pinned. Run in council gate + scheduled task.
  3. Config-diff gate: before deploy, snapshot service env keys + secret refs; compute post-deploy spec; if any key would be REMOVED → refuse unless --waive-config-removal "<reason>" given; waiver logged in deploy record. Uses --update-env-vars semantics, never --set-env-vars.
  4. Revision attribution: deploy applies labels/annotations wt-id, ticket, commit on the new revision; deploy record state/deploys/<app>/<revision>.json mirrors it. "Whose is 00034?" = rc whose <app> 00034.

3.4 Hooks + visibility

  • UserPromptSubmit hook: if session's registered worktree maps to a claimed app, emit claim status line (holder, ttl, queue) into context each prompt — peers see contention without asking.
  • Claims state repo syncs via existing fleet-repo-sync (add to manifest.json) — 30-min task is the fallback; rc claim/release push immediately, so freshness in practice = seconds.

3.4b Additions from WT-64ed survey reply (2026-07-30, WING-240 session — 13 incidents witnessed)

  • Traffic is a claimable intent (S1): add intent: traffic to the claim model. gcloud run services update-traffic (pin/unpin/to-latest) is a gated verb — two sessions changed prod traffic within one hour today without each other's knowledge. Deploy claim does NOT imply traffic claim.
  • Attribution-before-traffic (S2): refuse to route traffic to a revision lacking provenance (wt-id/ticket/commit labels or deploy record). Unattributed revision 00034 cost 3 sessions ~1h.
  • Remote cancel on abort (S3): rc build traps local kill (Ctrl-C/TaskStop) and issues gcloud builds cancel <id> server-side; build ID captured at submit. Kills zombie-build class (v1 scope now — today produced a real one, not rare).
  • Atomic invocation, no pipelines (S4): rc executes build/deploy itself as discrete checked steps — never as a shell chain. Exit codes checked per step; a failed build can never fall through to deploy (today: build && deploy | tail swallowed exit code → prod silently rolled back ~24h via mutable tag).
  • Serving-vs-latestReady honesty (S5): rc status <app> prints serving revision (traffic-weighted) and latestReady SEPARATELY, labeled. Conflation cost ~1h misattribution today.
  • Verification staleness binding (S6, ledger metadata): deploy/build records bind claims to digest/commit/ETag of what was verified, so drift (code/bucket/config moved after check) is detectable. v1: record refs; drift-alert tooling later.

3.4c Consolidated survey requirements (WT-9265, WT-8f26, WT-8ccc, WT-373f replies, 2026-07-30)

The five-session survey (~60 events; detail in FLT-182-baseline-incidents.md) converges on:

  • Provenance = in-project build record (S7): refuse to deploy ANY digest lacking a build record in this project (3.2's state/builds/). Blocks foreign images (Google hello-container promote near-miss, unattributed 00034) mechanically, without knowing who made them. Validity ≠ provenance.
  • Verify by artifact, never exit code (S8): gcloud exit codes lie (documented exit-0 failures; PowerShell $ErrorActionPreference ignores native exes). Post-deploy assertion set: latestCreatedRevisionName == latestReadyRevisionName (polled — service Ready=True is NOT a rollout check; it stays true on the old revision), env/secret key names+counts unchanged vs pre-deploy snapshot, served image digest ∈ expected repo, provenance labels present. Every rc step succeeds only by verified artifact.
  • Config-diff vs LIVE, not vs declaration (S9): 3.3.3's diff snapshots the LIVE service (declared files can omit live keys — env-ladder declared 6/19 and 0/7 of live keys on two services; derived/ injected keys make declarations misleading both directions). Mechanical live-vs-declared diff also runs in CI.
  • Tag reaping (S10): rc audit flags registry tags whose digest is stale vs newest build (removing a tag's publisher while the tag lingers makes tag-deploys progressively staler — reverse hazard).
  • Break-glass announces (S11): any waiver/break-glass fires worktree-mail to all live sessions + ledger entry. Incidents feel like they waive coordination; that is when it matters most.
  • Preflight assertions (S12, from WT-373f): porcelain-clean on exact ref; build from immutable SHA never branch/tag; runtime-write paths (e.g. wwwroot/.seeded class) all gitignored; merge-base fresh vs origin/master and diff feature-only; checkout can SEE the ref it claims to build (refspec check).
  • Content surface (S13, v1-lite): bucket/content publishes are a claimable intent (intent: content) so code deploy + content publish can't invalidate each other's verification unseen. Full content-provenance = follow-up ticket.
  • Ancestry / revert-by-deploy gate (S14, WT-570b): rc deploy refuses when the candidate commit is not a descendant of the currently-serving revision's commit (git merge-base --is-ancestor) — a branch that diverged from master would silently roll live work back. Override requires explicit --allow-rollback (which is the rc rollback path, B6). Unrelated-histories counts as non-ancestor (WT-ab3e's WING-227 lane was built on a duplicate project lineage; deploy would have reverted three live features). Build gate also refuses when another build for the same app/tag is in flight (same-tag double-build race, WT-ab3e/WT-9265).
  • CI-gate-runs check (S15, WT-570b): provenance gate verifies the commit's test gate actually ran in CI (check run/workflow presence for that SHA), not merely that a suite exists — "green locally, never wired into ci.yml" made every green claim CI-meaningless for a whole lane.
  • GCP surface extension (S16, Andrew 2026-07-30): gates extend beyond Cloud Run deploys to ALL release-relevant GCP mutations, via gcloud CLI wrappers (no GCP MCP exists in registry; CLI already the pattern): (a) Artifact Registry — gcloud artifacts docker tags add|delete and image deletes are gated verbs (claim intent build|deploy; tag adds refused on mutable non-sha tags per 3.3.2/S10); (b) GCS buckets — gsutil cp|rsync|rm and gcloud storage cp|rsync|rm targeting buckets registered in apps.json are gated verbs (claim intent content); rc publish <app> wraps bucket writes with pre/post object-generation capture (S6 staleness binding) and writes a content-publish record; (c) hook verb classes updated accordingly. Cloudflare/Porkbun APIs (available now) cover site-related DNS/CDN/domain ops (ww-site et al.): future intent: dns gated class, follow-up ticket — not v1, but apps.json schema gets an optional domains field now so registration is ready.
  • Environment awareness (S17, Andrew 2026-07-30): apps.json is an app × environment matrix — each app lists its rungs (qa, prod, video, etc.), each rung with its own service/project/region/ bucket. Claims, gates, provenance, deploy records, and the ledger are all scoped to (app, environment): claiming ww-site@qa never blocks ww-site@prod. Rung policy is declarative per-environment: prod-class rungs require waiver ack + ancestry gate + full post-verify; qa-class rungs may relax TTL and ancestry but NEVER the provenance/digest gates (the hello-container near-miss was a qa rung). rc promote <app> --from qa --to prod moves a digest between rungs through the full target-rung gate set, refusing placeholder/foreign images (S7) and running the live config diff against the TARGET rung (the WING-219 replace-semantics + 0/7-keys lessons). Integrates with, not replaces, the WING-219 env-ladder: env-ladder declares config; rc gates the operations that apply it.
  • Detector-effectiveness stat: WT-8f26's 22 incidents: only 6 caught by formal review gate; 8–10 catchable by mechanical live-diff + provenance checks. That is this tool's target class. Ledger source field tracks detector so this ratio is measurable post-tool.

3.5 Incident & near-miss ledger (rc incident, Kerr-aware — added per Andrew 2026-07-30)

Purpose: measurable baseline of how often release coordination goes wrong or nearly wrong, so the tool's effect is evaluable ("did FLT-182 actually help").

  • Ledger: append-only JSONL state/incidents/<app>.jsonl. Record: { utc, app, wtId, ticket, class, severity: incident|near-miss, source: gate|self|peer|retro, detail, commit/revision refs }. Classes seeded from the ticket's six: dirty-build, zombie-build, unattributed-revision, mutable-tag-pin, env-key-loss, land-race (+ other).
  • Auto-capture is primary: every gate refusal (claim denial, provenance refusal, digest refusal, config-diff refusal, hook deny) writes a source: gate near-miss automatically. No one has to choose honesty under deadline; the deny path IS the logger.
  • Self/peer reports: rc incident report --app X --class Y --severity near-miss "detail" for things gates can't see (chat-based races, zombie builds). Retro backfill: --utc flag allows historical entries; baseline seeded with the six 2026-07-30 incidents + peer-session survey replies (WT-8f70 collecting via session mail).
  • Kerr guard (folly of rewarding A while hoping for B): hoped-for B = fewer real failures; measurable A = ledger counts. Rules so A can't be gamed against B:
  • Ledger counts are NEVER a per-session or per-teammate score, never ranked, never surfaced as "who caused most incidents". Attribution exists for forensics, not leaderboards.
  • Gate-refusal near-misses are read as the TOOL WORKING (prevented class), reported separately from severity: incident (reached prod). Success metric = incidents-reached-prod trend, not total events logged. A rising near-miss count with flat incidents = healthy.
  • No target thresholds on logged counts (a target invites under-logging). rc stats <app> prints both series side by side with that framing baked into the output.

3.6 Council build-blockers (fable-tier council 2026-07-30, verdict: sound-to-build after B1–B7)

  • B1 Lease/CAS claim model (replaces §3.1 mutable queue array): claim record carries leaseId (GUID) + monotonic generation; every gated op/heartbeat/release must present matching leaseId, mismatch = hard fail; heartbeats never create/rebind. Queue = append-only per-claim event files keyed (wtId, nonce), state derived by replay; snapshot after CAS only. RMW protocol: pull --rebase → re-derive → re-apply → push, bounded retries w/ jitter. Heartbeat runs on background timer for the full child-process lifetime of builds/deploys (Cloud Build > 15-min TTL otherwise = day-one failure); timer failure fails the op. Eviction grace = ttl + 5 min, log both clocks. Release promotes queue head in the SAME commit, but promoted holder gets ~5-min acceptance window (first rc op converts to full lease; else skipped). Claim state lives in a dedicated release-coordinator-state repo/branch, not the tool's code repo. Exclusivity is app-level; intent is metadata.
  • B2 Token contract: RC_GATE_TOKEN = single-use, ≤60 s expiry, HMAC over (sessionId, app, verb-class + normalized argv fingerprint, nonce, expiry); hook stores hash only, matches actual command, consumes atomically.
  • B3 Two-layer enforcement: IAM is the enforcement layer — everyday ADC on all PCs is viewer-only on prod projects; deploy rights live in a per-app deployer service account only rc impersonates. Hook = ergonomics (fast friendly deny), NOT the security boundary (PreToolUse is Claude-harness-only; Cedric/Jenny/Gronktayvius harnesses bypass by construction). PATH shims optional defense-in-depth.
  • B4 Hook hardening: classify before any throwing I/O; outermost handler denies when classification itself fails (crash-parser must not fail open). Normalize executable paths / gcloud.cmd / cmd /c / pwsh -Command|-EncodedCommand; parse git refspecs + remote URLs, not argv substrings; docker-registry allowlist config-driven fail-closed; cached state on hot path (re-pull only on verb match); central denial log; residual surface (direct API clients, unhooked terminals) documented as known-open + rc audit scheduled.
  • B5 rc enforces internally: rc build/deploy verify claim+leaseId, registered worktree, provenance themselves — hook absence ≠ gate absence.
  • B6 Coverage: config-diff covers whole service spec (SA, VPC connector, secret refs, volumes, min/max instances), canonicalized; deploy refuses if attribution label apply fails; rc rollback <app> to prior digest through same gates (deploy record keeps previous revision); provenance records committed immediately + content-hashed (tamper-evident); state schema versioned (old binary refuses newer format); rc version pinned in OneDrive pool.
  • B7 Break-glass outside the hook (broken hook must not strand emergencies); solo-fleet unacked waiver pages Andrew via Pavlok (rate-limited 1/app/24h) + local console confirm — never silent.
  • Zombie builds v1 middle: record Cloud Build id in provenance; pre-deploy refuse while a concurrent build for the app is in flight (builds list --ongoing); best-effort builds cancel on SIGINT/session-end (no guarantee claimed). Full lifecycle cancel = follow-up ticket.

4. Non-goals (v1)

  • No daemon/server; git is the coordination bus.
  • No platforms beyond Cloud Run (interface IDeployTarget keeps door open).
  • No automatic merge/rebase execution — tool gates and attributes; humans/sessions still run git.
  • No zombie-build kill (server-side cancel) — v1 detects mismatch post-hoc via digest/provenance check; cancel API integration ticketed separately.

5. Failure semantics

  • Gates FAIL CLOSED: provenance/digest/config-diff refusal on any verification error.
  • Claim store unreachable (git push fails, offline): claim ops fail closed; deploys blocked. Emergency override --break-glass "<reason>" logs waiver record with ISO-8601 expiry ≤24h (mirrors council waiver policy). Kerr fix K3 — waivers need a second pair of eyes: --break-glass and --waive-config-removal take effect only after ack from a second live WT session (rc ack-waiver <id> from any other registered session; worktree-mail notifies peers). Solo-fleet fallback: if no other live session exists within 5 min, waiver proceeds but is flagged unacked: true and surfaced in the next council gate. Free-text-reason-only waivers are recorded but inert.
  • Hook must never crash a session: exceptions → allow with warning line (matches worktree-identity), EXCEPT deploy/build command matches, which deny on internal error (fail closed where it gates prod).

6. Testing/gates before land

xUnit suite (claim race via concurrent push simulation, TTL eviction, provenance refusals, config-diff matrix, hook JSON contract), vulnerable-package scan, /council-code-review with both security seats (tool gates prod deploys → security-sensitive), kerr-lens pass on this spec first.