Skip to content

Raft HA deploy runbook — shadow drill + live flip (Jira FLT-49, executed by WT-902b)

Andrew-ordered 2026-07-17: "prep the firewall commands and shadow cluster config and then run the drill and if it works than flip." This file is the exact procedure + the record of what ran.

Fleet facts (verified live 2026-07-17)

Role nodeId Host Tailnet IP Notes
Voter gaming DESKTOP-D249KD4 100.110.224.88 current live single-writer authority (UsageGovernorAdmitAuthority scheduled task, serve-admit :47610); ledger C:\Users\AndrewJonesDev\.claude\usage-governor\agy-ledger.jsonl
Voter greatroom ANDYGREATROOMPC 100.69.90.95 this box
Voter streaming DESKTOP-SCDT9HI 100.78.38.60
Observer bedroom ANDREWBEDROOMTV 100.85.245.95 least stable; runs the stateless forwarder only (raftPort 0)
  • All 4 boxes: .NET 10 runtime present, fleet key at ~/.claude/fleet-dispatch.key, fleet-dispatch listener reachable and elevated (admin=True verified on the 3 remotes).
  • Remote execution: FleetDispatch.exe send --target <PC> --kind pwsh --wait (payload = PowerShell).
  • Build artifact: %OneDrive%\PC-Bootstrap\_shared\usage-governor\ug-raft-a410dd5.zip (publish of usage-governor master a410dd5, which contains the gated Raft work c091939; SHA256 7412953E704EAEE92484BD45D938DCC77A6915ECCAE070D944B4BC029AE031BC).
  • Ports — real: raft TCP 47620, admit HTTP 47621. Shadow drill: 47630/47631 (separate UG_BASE_DIR=%USERPROFILE%\.claude\ug-raft-shadow, so the live ledger and live config are untouched by the drill).

Non-negotiable correctness requirements

  1. Identical agy policy config on all 3 voters. AgyRaftStateMachine.ApplyAsync re-runs AgyPolicy.EvaluatePoolGate on EVERY node from its own config + ledger view; divergent budgets/floors would make nodes book different outcomes for the same committed command. At flip time, gaming's live agy config block is the source of truth and is copied (with the raft block added, admitEndpoint removed) to every voter.
  2. Identical pre-Raft ledger history on all 3 voters. Policy totals read the ledger file, so voters must start from byte-identical copies of gaming's live ledger, SHA-256 verified, taken AFTER the old authority is stopped (no writes in flight).
  3. Order of the flip on gaming: stop+disable UsageGovernorAdmitAuthority FIRST, then edit config (raft.enabled=true) — serve-admit and serve-raft refuse to coexist (mutual-exclusion guard, both directions), and the ledger snapshot must be quiescent.
  4. Firewall posture: Windows inbound default is Block, so ONE scoped Allow rule per port is the whole story — raft TCP allowed only from the other two voters' tailnet IPs; admit HTTP allowed only from the tailnet CGNAT range (100.64.0.0/10). This closes RAFT-BLOCKERS #3's deploy gate (transport auth deferred; port reachability restricted to exactly the peers instead).

Phase 1 — permanent firewall rules (per voter; drill + real ports in one pass)

For voter X with peer voter IPs P1, P2 (run elevated, via dispatch):

# raft TCP: only the two peer voters may connect
New-NetFirewallRule -DisplayName "ug-raft-peers-in" -Direction Inbound -Action Allow `
  -Protocol TCP -LocalPort 47620,47630 -RemoteAddress <P1>,<P2> -Profile Any
# admit HTTP: tailnet only
New-NetFirewallRule -DisplayName "ug-raft-admit-in" -Direction Inbound -Action Allow `
  -Protocol TCP -LocalPort 47621,47631 -RemoteAddress 100.64.0.0/10 -Profile Any

Bedroom (observer) gets only the admit-HTTP rule (it has no raft port).

Phase 2 — shadow cluster

Per voter, via dispatch (elevated listener):

  1. Unzip ug-raft-a410dd5.zip → %USERPROFILE%\.claude\ug-raft-shadow\bin\ (SHA-256 verified against the value above after OneDrive sync).
  2. Write %USERPROFILE%\.claude\ug-raft-shadow\config.json:
{
  "agy": {
    "geminiBudget5hWeighted": 10000000,
    "geminiBudgetWeeklyWeighted": 100000000,
    "raft": {
      "enabled": true,
      "nodeId": "<gaming|greatroom|streaming>",
      "voting": true,
      "raftBind": "0.0.0.0:47630",
      "httpBind": "+:47631",
      "peers": [
        "gaming=100.110.224.88:47630:47631",
        "greatroom=100.69.90.95:47630:47631",
        "streaming=100.78.38.60:47630:47631"
      ]
    }
  }
}
  1. Start detached: $env:UG_BASE_DIR="$env:USERPROFILE\.claude\ug-raft-shadow"; Start-Process -WindowStyle Hidden "$env:UG_BASE_DIR\bin\UsageGovernor.exe" -ArgumentList "serve-raft".
  2. Verify each node's GET http://<ip>:47631/health answers and GET /raft-status shows exactly one leader cluster-wide.

Phase 3 — the drill (the RAFT-BLOCKERS #4 scenario on real machines)

  1. Baseline admit through the leader (bearer = fleet key; use the shadow-config agy-admit from this box) → allowed; verify the spend line lands in all 3 shadow ledgers (%USERPROFILE%\.claude\ug-raft-shadow\agy-ledger.jsonl) with the same raftIndex.
  2. Partition the leader box (temporary block rules ON THE LEADER, both directions — outbound default-allow means the cut needs explicit blocks):
New-NetFirewallRule -DisplayName "ug-raft-drill-cut-out" -Direction Outbound -Action Block `
  -Protocol TCP -RemotePort 47630 -RemoteAddress <P1>,<P2> -Profile Any
New-NetFirewallRule -DisplayName "ug-raft-drill-cut-in" -Direction Inbound -Action Block `
  -Protocol TCP -LocalPort 47630 -RemoteAddress <P1>,<P2> -Profile Any
  1. Assert: admit against the isolated leader's :47631 → refused (raft fail-closed mode, and nothing booked anywhere for that attempt); the 2 majority voters elect a new leader and an admit against them → allowed.
  2. Heal: Get-NetFirewallRule -DisplayName "ug-raft-drill-*" | Remove-NetFirewallRule.
  3. Assert: old leader rejoins as follower; all 3 shadow ledgers converge to the same raftIndex set (baseline + during-partition admits; the isolated attempt appears NOWHERE).
  4. Tear down: kill the 3 shadow processes. Shadow dirs kept for the record.

Go/no-go: every assertion in step 3 + 5 must hold, verified by ledger artifact read off each box — not by HTTP responses. Any failure = NO FLIP, collect logs, report.

Phase 4 — the flip (only after a green drill)

  1. Gaming: Stop-ScheduledTask UsageGovernorAdmitAuthority; Disable-ScheduledTask ... then SHA-256 the live ledger and copy it + the live agy config block to %OneDrive%\PC-Bootstrap\_shared\usage-governor\ (migration payload).
  2. Greatroom + streaming: after sync, hash-verify and install the ledger copy at %USERPROFILE%\.claude\usage-governor\agy-ledger.jsonl (backing up any existing file first).
  3. All 4 boxes: merge into live ~/.claude/usage-governor/config.json: gaming's agy policy values, admitEndpoint REMOVED, plus the real raft block (ports 47620/47621, peers list with bedroom=100.85.245.95:0:47621 included, voting:false + observer role on bedroom).
  4. Unzip build → %USERPROFILE%\.claude\usage-governor\bin-raft\ on all 4; register scheduled task UsageGovernorRaftAuthority (at startup, run as the box user, hidden) running serve-raft; start it.
  5. Verify: one leader on /raft-status; real agy-admit from this box books exactly once on all 3 voters; observer forwards.
  6. Watch the first hour of admits (each box's agents hit it organically).

Rollback (any step)

  1. Stop + disable UsageGovernorRaftAuthority everywhere.
  2. Re-enable + start UsageGovernorAdmitAuthority on gaming (its ledger was never moved — the copies went outward; worst case replay-safe because raft entries carry AdmitId/RaftIndex).
  3. Restore raft.enabled=false + admitEndpoint=http://100.110.224.88:47610 in every box's config.
  4. The firewall Allow rules are harmless to leave; drill cut rules must be removed (step 4 heal).

Execution log (2026-07-17 evening, WT-902b)

Phase 1+2 (shadow): setup dispatched to 3 voters ~18:56 CDT — zip hash verified on each, scoped firewall allows applied, shadow nodes started. Cluster elected immediately (leader=gaming, term 8, all 3 agree). Baseline admit via real client → allowed; spend|baseline-1|idx=2 identical on all 3 shadow ledgers.

Phase 3 (drill): GREEN. - 18:57:55 partition applied on gaming (block rules, both directions, port 47630). - Isolated live leader answered the authed direct admit with raft: gaming is not leader and no leader is currently known; retry another peer — genuine Raft fail-closed, nothing booked. - Majority re-elected around it (term 8→9, streaming) and served during-partition. - 18:59:13 healed; gaming rejoined as follower; final term 13, one leader. - Convergence: all 3 shadow ledgers = {baseline-1@idx2, during-partition@idx6}; asked-in-minority present NOWHERE. Exactly-once held on real machines. - Finding (availability, not correctness): AdmitViaPeersAsync treats a live node's "no leader known" answer as final for the attempt — with the isolated node first in the peer list the client fails closed even though the majority is serving. Money-safe; follow-up ticket recommended (skip-to-next-peer when a node reports no leader and unasked peers remain).

Phase 4 (flip): DONE. - Old authority UsageGovernorAdmitAuthority on gaming STOPPED + DISABLED ~19:05 CDT. Live ledger snapshot: 23 lines, SHA256 EC0174B4…F38, distributed via OneDrive and hash-verified before install on greatroom + streaming (both installed byte-identical). - Gaming's live config had NO agy block → policy parity across voters = AgyConfig defaults; each box's config gained the real raft block (47620/47621, bedroom observer :0:), admitEndpoint removed, original config backed up as config.json.pre-raft.bak. - UsageGovernorRaftAuthority scheduled task (serve-raft, AtStartup, S4U, RunLevel Highest, restart 3×1min) registered + started on all 4 boxes; binaries at ~\.fleet\usage-governor-raft\. - INCIDENT: greatroom's first serve-raft process wedged (accepted TCP on 47620/47621 but never answered — HTTP hung, raft handshakes never completed), which poisoned every election round: gaming/streaming climbed 150+ terms with no winner. Kill + task restart fixed it; cluster elected within seconds of the healthy restart. Watch item: if a node wedges again, restart its task — and consider a health-probe watchdog on the task. - Post-flip verification: one leader; real agy-admit from greatroom → ALLOWED with the migrated history counted (weekly 83,384.5 weighted from the 23 pre-raft lines); admit via bedroom observer → ALLOWED; ledgers on all 3 voters = 25 lines, identical, flip-verify-1 and via-observer-1 each booked exactly once. - Watch item: term churn observed in bursts (281→330 over ~15 min) settling to stable stretches — consistent with sub-second leader stalls under host load against the 150–700ms election bands. Fail-closed blips only; recommend a follow-up ticket to raise prod election timeouts (e.g. 4–10× the test bands — failover SLA here is minutes, not milliseconds).

Rollback state: old task still present (disabled) on gaming; every box has config.json.pre-raft.bak and (greatroom/streaming) agy-ledger.jsonl.pre-raft.bak. Rollback = runbook section above, ~2 minutes via dispatch.

Phase 6 (watchdog + peer-walk + retro fixes rolled out, 2026-07-18): the 3 follow-up chips from phase 5 (wedge watchdog, E3 retro review, client peer-walk) plus a peer session's flaky-test fix all committed real work, but 4 of those 5 commits — plus a further "harden watchdog per council review" pass — never got PUSHED from the ug-wt-raft worktree (found + recovered: the hardening pass was sitting fully written but uncommitted on disk). All landed on usage-governor master 9c5ca6e (one merge conflict in GovernorPaths.cs, purely additive, both sides kept; 464/464 tests green). Rebuilt + rolled the REAL build (hash D27B0BF1…, confirmed watch-raft: ok locally before shipping) to all 4 boxes. UsageGovernorWatchRaft (1-min probe-and-cure task) registered and confirmed Ready on gaming/streaming/bedroom.

Known gap: greatroom's UsageGovernorWatchRaft task is NOT registered — creating a NEW scheduled task with RunLevel Highest requires the caller to already be elevated, and neither local Bash-tool PowerShell sessions on greatroom nor self-targeted fleet-dispatch (governor-gated locally, was RED throughout this session) could provide that. Existing-task start/stop/run (used for the authority binary swap) does NOT require caller elevation — only NEW task creation does — which is why the authority update succeeded unelevated but this didn't. Greatroom's UsageGovernorRaftAuthority itself IS on the current build and healthy; it just has no automated wedge-recovery yet. Follow-up needed: one elevated PowerShell command on greatroom (copy the Register-ScheduledTask block from Phase 2/rolling-update2.ps1's watchdog section), or wait for the local governor to clear and retry via FleetDispatch.exe send --target ANDYGREATROOMPC.

Separately flagged: the worktree-identity CLI's documented worktree claim/release/list sub-verbs are broken in the installed version (System.ArgumentException: Unexpected positional argument) — real tool bug, hit while trying to release a stale cross-session worktree lock; worked around by waiting for the lease to expire naturally rather than fixing the tool mid-deploy.

Phase 5 (post-flip hardening — watch item #2 closed): added agy.raft.electionTimeoutBaseMs (all Raft timing scales linearly from it; default 150 = original test tuning, byte-identical) — usage-governor master 5a365d3, full suite 273/273. Rolled the new build + electionTimeoutBaseMs=1500 to all 4 boxes via dispatch (hash-verified zip 2383B7AE…). Before: 150ms bands churned 79 elections in 10 min (term 378→457, each a fail-closed blip). After: term 457 held FLAT across a 9-minute sample (0 elections); admit still ALLOWED, migrated history intact. Watch items #1 (wedge watchdog) and #3 (client peer-walk skip-to-next) remain open as follow-up tickets — neither is a correctness risk.

Watch item #1 closed — serve-raft watchdog

Install registers a one-shot watchdog every minute on every box. The verb safely no-ops when agy.raft.enabled=false:

schtasks /Create /SC MINUTE /MO 1 /TN "UsageGovernorRaftWatchdog" /TR "$exePath watch-raft" /F

Deploy through fleet dispatch using the Phase 4/5 per-box install pattern, then verify with Get-ScheduledTask -TaskName UsageGovernorRaftWatchdog. For a manual bin-raft deployment, dispatch this after expanding the build (replace path for each box):

schtasks /Create /SC MINUTE /MO 1 /TN "UsageGovernorRaftWatchdog" /TR '"C:\Users\<user>\.fleet\usage-governor-raft\UsageGovernor.exe" watch-raft' /F

Alternative: invoke watch-raft after scan in the existing five-minute task. Keep it one-shot; do not create a long-lived watchdog process. Dedicated one-minute task is primary deployment.

Watch item #3 remains open; watch item #1 is closed by FLT-49.