Skip to content

RAFT-BLOCKERS — dotNext API gaps and the design decisions they forced (FLT-17)

Per the spec's guardrail: "If you hit a genuine fork (e.g. dotNext API can't express something the design needs), STOP and leave a clear options note — do not guess in the money path." This file is that note. None of these block the invariant (single writer, exactly-once admit); they're places where the literal spec wording and dotNext 6.4.1's actual capabilities diverge, and what was chosen instead, verified against the actual dotNext source (cloned dotnet/dotNext at tag 6.4.1, not recalled from memory) before writing a line of the money-path code.

1. No non-voting "learner" membership — the observer is NOT a Raft cluster member

Spec wording: "3 VOTING nodes + 1 NON-VOTING observer... Quorum = 2 of 3, so only 2 machines need be up... The 4th PC is an observer that can serve reads/forward but does not vote."

What dotNext actually has: a per-node Standby flag (IClusterMemberConfiguration.Standby, RaftCluster.TcpConfiguration.Standby). Verified against src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs and StandbyState.cs: Standby only prevents the LOCAL node's own state machine from ever transitioning to Candidate. It does not remove that node from the cluster's membership count. Every node in IClusterConfigurationStorage counts toward the majority denominator (RaftCluster.cs, LeaderState.cs — majority is always members.Count / 2 + 1 over the full configured set). There is no "learner" or non-voting-replica concept anywhere in DotNext.Net.Cluster — grepped the whole tree for Learner/NonVoting and found nothing.

Why that's a real fork, not a nit: if the observer joined as a Standby=true member, the cluster would have 4 configured members and need 3 of 4 for majority — directly contradicting "quorum = 2 of 3, so only 2 machines need be up." A Standby observer would make the deployment less available (needs 3 machines up instead of 2), which is the opposite of the stated goal.

Decision: the observer runs no Raft cluster member at all. It is a stateless HTTP forwarder (AgyRaftObserverServer) that proxies every /admit to the 3 voting peers using the exact same peer-list-plus-leader-redirect logic the CLI client uses (AgyAdmitClient.AdmitViaPeersAsync). The Raft cluster's static membership config (AgyRaftPeerList.VotingSorted) never includes the observer, so quorum math is exactly "2 of 3" regardless of whether the observer's process is up, down, or doesn't exist. This also makes "observer never leads / never votes" trivially, structurally true (there's nothing to disprove — it never runs the Raft protocol), rather than something a test has to catch dotNext from allowing.

Residual risk: the observer has no local replicated copy of the ledger for local reads — every read (a future agy-status-over-raft) it serves would also have to be a forward. Not needed for this pass (no read verb was in scope); flagging so a future "let the observer answer reads locally" task doesn't assume it has state it doesn't have.

2. WAL compaction is not implemented — full log retained, unbounded growth

What's in place: AgyRaftStateMachine.ApplyAsync always returns false, which tells SimpleStateMachine never to trigger PersistAsync/a snapshot. This was a deliberate choice, not an oversight — tracing through WriteAheadLog.cs (appliedIndex = snapshotIndex at construction, where snapshotIndex comes from the most recent on-disk snapshot file) showed that if PersistAsync ever ran and actually depended on it, a node restart would resume applying from that snapshot's index instead of from 0 — but the snapshot only records what PersistAsync chose to serialize. Since this state machine's real state is agy-ledger.jsonl (AgyLedger.Append, fsync'd per entry) and NOT anything PersistAsync writes, a rejoining node's snapshot-based catch-up would have no way to recover ledger CONTENT for the truncated portion of the log if compaction had ever run. Keeping the full log (no compaction) means a rejoining/catching-up node always recovers by replaying every committed AdmitCommand through ApplyAsync again, which is made safe by the AgyLedgerEntry.RaftIndex idempotency check (see AgyRaftStateMachine's class doc) — this is also exactly what makes crash-restart safe for a single node, not just multi-node catch-up.

Consequence: the Raft WAL directory (~/.claude/usage-governor/raft/<nodeId>/wal) grows without bound — every historical committed command stays in the log forever. Not a correctness problem (never causes a double-spend or a lost spend); it is a real operational concern for a long-lived production deployment (disk growth, and a freshly-added node would need to replay an ever-larger history to catch up).

Recommendation before a long-term production rollout: implement PersistAsync/RestoreAsync to actually serialize+restore the applied ledger state (or an index cursor + a way to fetch the authoritative ledger content out-of-band on rejoin), so ApplyAsync can safely return true periodically and let dotNext truncate the log. Out of scope for the Sunday HA-correctness deadline; flagging so it isn't silently forgotten.

3. No transport-level auth on the Raft TCP port in this pass

Spec wording: "add a shared-secret/mTLS gate on the Raft port so a stranger can't join the cluster."

What's in place: RaftCluster.TcpConfiguration.SslOptions genuinely supports mTLS (confirmed against the working RaftNode example at the pinned tag, UseTcpTransport(..., useSsl: true) + SslOptions.ServerOptions/ClientOptions + RemoteCertificateValidationCallback), so this is not an API gap — dotNext can express it. It was deferred rather than guessed at under time pressure, because doing it properly needs real per-node certificates (not a reused dev self-signed one shared across all 4 machines, which would defeat the point), and because a stranger reaching the Raft TCP port cannot actually join the cluster in this design regardless: membership is static (ColdStart = false, no Announcer, no dynamic add) — an unauthenticated peer can send malformed frames but there is no join path for it to exploit into becoming a 4th voting member or a leader.

Mitigation until this is built: restrict the Raft TCP port (agy.raft.raftBind, default :47620) at the network layer — tailnet ACL / Windows Firewall inbound rule allow-listing exactly the 3 voting peer IPs — before this is deployed to the real fleet. This does NOT block the switchover-ready build (client-facing /admit already has the existing FleetKeyAuth bearer gate, which is the actual admission-authority trust boundary); it blocks fleet deployment, which the spec explicitly gates separately ("Deploy is a separate gated step Andrew approves").

4. In-process TCP partition simulation can't isolate a proactively-dialing node (e.g. a leader)

Spec wording (task 6): "Partition the leader into the minority — it CANNOT commit; the majority elects a new leader and keeps serving; when the partition heals, the old leader's uncommitted entries are discarded."

What was attempted: a test-only LoopbackProxy (tests/UsageGovernor.Tests/Raft/LoopbackProxy.cs) sits in front of each voting node's real TCP port; every other node's static peer config addresses it via the proxy port, so blocking the proxy refuses new inbound connections and force-closes active ones. The first version of the partition test blocked the CURRENT LEADER's proxy and expected the other two to elect a new leader. It didn't work — confirmed empirically, not just theorized: a diagnostic run showed the two survivors dueling through 50+ terms over 40+ seconds without ever electing (the fix for that dueling-candidate problem is documented above the LowerElectionTimeout/UpperElectionTimeout staggering in AgyRaftNode.Create), and even after fixing the dueling-candidate bug, the survivors STILL never noticed the "isolated" leader was gone.

Root cause: dotNext's Raft transport is connection-oriented and a leader PROACTIVELY DIALS OUT to send heartbeats/AppendEntries. That outbound connection is initiated by the leader, targeting the OTHER nodes' proxies (which are not blocked) — it never touches the leader's OWN inbound proxy at all. So "block the leader's inbound proxy" cuts off new connections INTO it but does nothing to stop it from successfully continuing to talk OUT to everyone else. The followers kept receiving real heartbeats the entire "partition" and correctly never started an election. This is not a dotNext bug — it's a real limit of destination-only TCP proxying against a transport where any node can freely redial: genuine per-link blocking would require either the receiving node to distinguish traffic BY SOURCE (not possible over 127.0.0.1 with ephemeral client ports without protocol-level sniffing) or intercepting the DIALING node's own egress (dotNext's RaftCluster.CustomTransportConfiguration takes a pluggable IConnectionFactory that could in principle do this, but implementing a working custom connection-oriented transport was out of scope for this pass).

What's tested instead: Follower_partition_majority_keeps_serving_isolated_follower_catches_up_on_heal isolates a FOLLOWER (which never proactively dials while healthy, so severing its inbound path is a real, faithful cut) while the leader + remaining follower — still a full 2-of-3 majority — keep serving without interruption; the isolated node catches up cleanly on heal with no double-booking (verified by the same converged-ledger artifact check every other test uses). This exercises real partition-tolerance and healing, just not the specific "majority re-elects around a still-alive isolated leader" sub-case.

RESOLVED (2026-07-17): the custom IConnectionFactory-based harness recommended below was built and the missing case is now an executed fault-injection test. AgyRaftNode.Create gained a test-only customTransport seam (dotNext RaftCluster.CustomTransportConfiguration), and tests/UsageGovernor.Tests/Raft/BlockableEgress.cs supplies an outbound-connection factory (delegating to Kestrel's socket transport) whose egress can be blocked at runtime — the side a destination-side proxy structurally cannot reach. AgyRaftJepsenTests.Live_leader_partitioned_into_minority_cannot_commit_majority_reelects_then_heal_converges severs a still-running leader's egress AND inbound (minority of 1), asserts it was still leader at the cut, that it refuses via a genuine Raft fail-closed mode (not-leader / propose-timeout / quorum-unreachable — explicitly excluding the "committed but apply unconfirmed" variant, which would mean a commit escaped the partition), that the majority elects a new leader and keeps serving, and that heal converges with the minority attempt booked NOWHERE (ledger artifact, all 3 nodes). Negative-controlled three ways: dev-run with no block (went red), dev-run with the historical inbound-only block (went red — leader kept committing over its own outbound dials, reproducing exactly the failure documented above), and an executable NEGATIVE_CONTROL test proving the unblocked custom transport serves and the core assertion can fire. The paragraph below is kept as history of why the original proxy-only attempt failed.