Skip to content

Raft HA for the agy admission authority — build spec (FLT-17)

Deadline: Sunday 2026-07-19. Target = a switchover-READY build (green tests, reboot-tested locally), not necessarily deployed. Deploy is a separate gated step Andrew approves.

Branch: wt/WT-0026/FLT-17-raft-ha off f39a106. Worktree isolation: build/test only here.

Why this exists (do not redesign — council already decided this, 3 seats, 2026-07-17)

The agy admission authority (AgyAdmitServer) meters paid-CLI spend against ONE ledger so the fleet stops oversubscribing every pool ~4x. Today it is a SINGLE authority on the gaming PC that FAILS CLOSED if that host is down. The owner now requires HA: it must survive one node dying and have another take over, because he uses it while away from home.

Council verdict (binding): - In-process C# Raft via dotNext.Raft (DotNext.Net.Cluster). NOT Kubernetes, NOT Swarm (their control planes need quorum across the SAME rebooting boxes = inherit the failure), NOT Postgres (couples to claude-mem; operational fencing can dual-primary). - Raft is chosen because it prevents two writers MATHEMATICALLY — a minority partition cannot commit. Postgres's fence-that-can-fail was explicitly rejected for this reason. - Topology: 3 VOTING nodes + 1 NON-VOTING observer. Quorum = 2 of 3, so only 2 machines need be up (matches measured uptime: gaming 7.6d, others 0.8–1.4d). The 4th PC is an observer that can serve reads/forward but does not vote (prevents a 4-node cluster needing 3 up). - Split-brain = silent double-spend = the exact 4x undercount this whole feature removed. So correctness dominates speed here. This is the one place to be slow and certain.

THE INVARIANT (every test exists to defend this)

At most one process, fleet-wide, ever appends to the ledger for a given logical time, and every admitted agy-admit is booked EXACTLY ONCE. No partition, reboot, or leader change may produce two writers or a lost/duplicated spend.

Design

The admission decision becomes a replicated state-machine command, not a local file append:

  1. A client's agy-admit (or the HTTP /admit) reaches ANY node. Non-leaders forward to the current leader (or return a redirect the client follows). Only the leader proposes.
  2. The leader proposes an AdmitCommand { model, estTokens, consumer, session, originHost, ts } to the Raft log.
  3. The command is committed only after majority replication (2 of 3 voters).
  4. On commit, EVERY node applies it to its replicated copy of the ledger state (the same AgyPolicy.EvaluatePoolGate + append). The evaluate-and-charge happens inside the committed state transition, so the decision is identical on every node and cannot split.
  5. The leader returns allow/deny to the caller AFTER commit. A proposal that never commits (leader lost quorum mid-flight) returns an error → client fails closed and retries against the new leader. Never a local fallback append.

Key point: the atomic reserve moves from FileShare.None on one disk to Raft consensus across nodes. The ledger file on each node becomes a materialized view of the committed log.

Reuse, do not rewrite

  • AgyPolicy.EvaluatePoolGate, AgyLedger.MakeSpend, AgyModels, AgyConfig — the policy is unchanged. Only the STORAGE/coordination changes.
  • FleetKeyAuth bearer stays for client→cluster auth. Raft inter-node traffic rides the tailnet; add a shared-secret/mTLS gate on the Raft port so a stranger can't join the cluster.
  • serve-admit keeps its split-brain guard for the LEGACY single-authority mode; add a new serve-raft verb (or a --raft flag) so the two modes never run at once on a host.

Config additions (AgyConfig)

"raft": {
  "enabled": false,                 // default OFF — legacy single-authority still works
  "nodeId": "gaming",               // stable per host
  "voting": true,                   // false on the observer (bedroom, least stable)
  "bind": "0.0.0.0:47620",
  "peers": [ "gaming=100.110.224.88:47620", "streaming=...:47620", "greatroom=...:47620",
             "bedroom=...:47620 (observer)" ]
}

Tasks (in order; each ends green)

  1. Add DotNext.Net.Cluster (Raft) package. Confirm the version targets net10. Build clean.
  2. Persistent log + state machine. Implement the Raft PersistentState whose committed commands are AdmitCommands; applying one runs the policy gate and appends to the ledger view. Persist Raft log + node identity under ~/.claude/usage-governor/raft/ (per node).
  3. Leader-only admit + forwarding. /admit on a follower forwards to (or redirects to) the leader. Clients discover the leader from any node.
  4. serve-raft verb wiring config → cluster startup. Mutually exclusive with serve-admit on the same host (guard both directions).
  5. Client change: AgyAdmitClient learns to try the peer list and follow leader redirects, still failing CLOSED, still fast-fail (2s connect) on a dead node.
  6. TESTS — this is half the ticket. codex was explicit: treat the library as critical, add Jepsen-style tests, NEVER expose a serve-without-quorum mode. Cover:
  7. 3-node happy path: N concurrent admits from all nodes → N ledger entries, no dup, no loss.
  8. Leader kill mid-admit: uncommitted proposal → caller error (no charge); committed → applied once.
  9. Partition the leader into the minority → it CANNOT commit; the majority elects a new leader and keeps serving; when the partition heals, the old leader's uncommitted entries are discarded (no double-spend).
  10. Reboot a node: it rejoins as follower and catches up from the log; totals still correct.
  11. Quorum loss (2 of 3 down): the cluster REFUSES to serve (fails closed), never serves without quorum.
  12. Observer node: forwards/serves reads but never becomes leader / never votes. These can run in-process with N cluster instances on loopback ports; no real reboot needed for CI.
  13. Local reboot/switchover drill doc: how to flip raft.enabled on, migrate the current single-authority ledger (SHA-256 verified, same procedure already used for the gaming move), and roll back to single-authority if Raft misbehaves.

Guardrails

  • C#-first. .NET 10. Match the existing UsageGovernor.Core style.
  • Every claim verified by artifact, never by exit code — ledger line counts + totals, not "tests passed". Negative-control the key tests (break the invariant, watch the test go red).
  • Do NOT deploy to the fleet. No SSH to peers, no scheduled-task changes, no touching the live gaming authority. Build + test in this worktree only. Andrew gates deployment separately.
  • Do NOT reboot any machine. Reboot scenarios are simulated in-process.
  • Council gate (/council-code-review) before it is considered land-ready. Flag every place the invariant could break for that review.
  • If you hit a genuine fork (e.g. dotNext API can't express something the design needs), STOP and leave a clear options note — do not guess in the money path.