Skip to content

FLT-113 + FLT-132 build report

Governor circuit breaker (FLT-113) plus the hostile-spawner test harness (FLT-132), built against the frozen FLT-133 contract at eab1c7a (src/UsageGovernor.Core/Contracts/Autonomy/, docs/autonomy-contract.md). Worktree wt/WT-6022/FLT-113, branched from origin/master. Not pushed, not merged.

This is a skeleton + proof harness, not the auto-spawn engine. FLT-115 (auto-spawn) is not built. Nothing added by this ticket is wired into the running governor CLI, ScanService, or the Raft admission authority (AgyRaft*) -- see "Proof nothing is wired in" below.

What was built

FLT-113: src/UsageGovernor.Core/Autonomy/

File Purpose
AutonomyBreakerOptions.cs Tunables not fixed by the FLT-133 wire freeze (storm window/threshold, retry-after advisory, spend windows). Deliberately outside Contracts.Autonomy -- the contract left "what the real gate call looks like" to FLT-113.
IFreezeCommandSink.cs Where the breaker signals a freeze. It does not enforce one -- enforcement is FLT-114, not built. A null sink is valid; the breaker still updates its own FreezeState and audit log regardless of whether anything is listening.
IAutonomyBreaker.cs The functional single-gate-wire interface (extends the Contracts.Autonomy.IAutonomyGatedAction marker). This is the ONE chokepoint every future spawn/spend path must call through. AutonomyBreaker is its only implementation; the only caller anywhere in this repo is the FLT-132 test harness.
AutonomyAuditLog.cs Append-only, thread-safe, per-subject-chained log implementing the ordering + reconstructible-offline requirement on AutonomyAuditEvent (contract sec 5). AutonomyBreaker is its only writer.
AutonomyBreaker.cs The breaker itself. Pure, in-memory, deterministic-clock-driven (every method takes an explicit DateTime nowUtc; no DateTime.UtcNow anywhere in the file, mirroring the ScanService.Run(recordLimitHit, now) injected-time convention already used in this repo).

Breaker design

Three governed surfaces, all against frozen FLT-133 types:

  1. Concurrency (TryAcquire/Release): enforces ConcurrencySemaphore.CapByTier against the ONE fleet-wide semaphore key. The breaker's own CurrentTier (settable via SetTier, meant to be driven by this repo's TierPolicy.Compute in a future wiring) is authoritative at grant time, not the requester's claimed AcquireRequest.Tier -- per the contract note that the breaker decides whether a claimed tier is still current. Over-cap requests are Deferred (skip-and-retry, SlotId = null, advisory RetryAfterSeconds) -- there is no queue data structure anywhere in the type; a rejected request leaves no trace beyond an audit Deny entry and, if part of a burst, a timestamp in a short pruned window used only for storm detection. Grants are tracked in a plain slotId -> requesterId dictionary; Release is idempotent (Dictionary.Remove on an unknown/already-released key is a safe no-op).

  2. Concurrency-storm detection (fork-bomb gate): a rolling window (ConcurrencyStormWindowSeconds, default 5) counts Deferred outcomes; once ConcurrencyStormDeferralThreshold (default 10) is hit inside that window, the breaker self-trips a FreezeCommand { Stage = FreezeSpawn } -- deny new spawns, leave already-granted work alone. This is a FLT-113 tunable, not a contract type: the contract's own BreakerHaltReason enum is scoped to spend-window trips and telemetry loss (sec 3), so a concurrency-storm trip is recorded as an AutonomyEventKind.Trip audit event with plain string Details, not forced into BreakerTripThreshold's spend-specific shape.

  3. Spend velocity (RecordSpend): sums SpendSamples per SpendWindowKind against configured SpendWindowDefinitions, independently per kind (contract sec 3 -- not a combined score). A trip pins a BreakerTripThreshold snapshot into the audit Details and halts with BreakerHaltReason.ThresholdTrip, issuing FreezeCommand { Stage = FreezeAll } (money is actively burning -- pause running work too, not just new spawns).

  4. Telemetry-loss fail-closed (Heartbeat/Tick, checked at the top of every TryAcquire and RecordSpend): a registered source (TelemetryHeartbeat.SourceId) that goes silent longer than its own declared MaxAgeSeconds halts with BreakerHaltReason.TelemetryLossHalt -- HALT, not "assume zero spend" -- and issues FreezeCommand { Stage = FreezeAll }, exactly as blocking as a real spend trip (same freeze stage, same subsequent denial of every TryAcquire). Staleness is measured against the breaker's own call-time nowUtc, not the source's self-reported EmittedAtUtc, so a compromised/confused source cannot claim freshness it didn't earn.

  5. Freeze / resume / kill: ApplyFreezeCommand lets an external authority (operator today, FLT-114 later) push a FreezeCommand directly; Kill sets the terminal killed state (FreezeCommand.Kill is only valid alongside FreezeAll, enforced by throwing otherwise, per contract sec 2). Resume clears back to Running and throws if the breaker was killed (a killed actor is replaced, not revived -- same pattern as taint's cleared-only-by-re-mint rule). Freeze is pause, not drain: neither the internal trip path nor Resume ever touches _grantedSlots -- a slot granted before a freeze stays granted through it, so "no double-slot / no orphan" holds by construction, not by a special case.

  6. Taint gating: TryAcquire reads AcquireRequest.Taint and denies outright (taint-spawn-blocked) for Tainted/Contaminated, per TaintDemotionRule.Table's SpawnBlocked column (contract sec 4). Full taint-transition enforcement (clamp-to-safe, cannot-cross-floor, transitive propagation on spawn) is FLT-114's job; the breaker only enforces the one column that gates its own surface (concurrency admission).

  7. Audit: every acquire outcome (Spawn on grant, Deny on deferred/denied), every trip (concurrency-storm, spend, telemetry-loss), every freeze/resume is appended to AutonomyAuditLog with a monotonic SequenceNumber, a synthesized EventId (evt-{n}), and a PriorEventId chained per SubjectId (the requester id for acquire-side events, the semaphore key fleet-autonomy/v1 for breaker-global events). The log is reconstructible offline: every PriorEventId resolves to an entry already in the log, with no external "current state" needed.

FLT-132: tests/UsageGovernor.Tests/Autonomy/

File Purpose
HostileSpawner.cs TEST-ONLY adversarial driver. TestOnlyHardOff = true constant documents the hard-off explicitly (on top of "it only exists inside the test assembly"). Never runs anything real -- drives IAutonomyBreaker with fake requester/source ids over a caller-controlled virtual clock. ForkBomb(...) hammers TryAcquire; SpendBurst(...) feeds RecordSpend. No Thread.Sleep, no DateTime.UtcNow.
RecordingFreezeSink.cs Test double for IFreezeCommandSink -- captures every FreezeCommand the breaker signals.
AutonomyBreakerHostileTests.cs The five enable-gates, as xUnit facts.

Gate results (all green, dotnet test)

# Gate Test(s) Threshold / assertion
1 Fork-bomb cap Gate1_ForkBomb_1000ConcurrentAcquires_CapHoldsExcessRejectedFreezeTripsFast 1000 concurrent TryAcquire at tier YELLOW (cap 6): exactly 6 Granted, remainder Deferred/Denied, zero over-cap slots ever held. Freeze (FreezeSpawn) trips by request index <=50 (storm threshold 10 deferrals inside a 5s virtual window) -- models "<=5s" as a deterministic tick budget. Exactly one freeze signal (no repeat-trip spam).
2a Spend-velocity Gate2_SpendVelocity_BurstAboveThreshold_TripsWithinBudget 5 x $2/tick = $10 against a $5/60s DollarsPerMinute threshold trips ThresholdTrip and issues FreezeAll.
2b Telemetry-loss fail-closed Gate2_TelemetryLoss_MissedHeartbeat_FailsClosedHalt_NotAssumeZeroSpend One heartbeat at t=0 (MaxAgeSeconds=90), none after; virtual clock advanced to t=200s trips TelemetryLossHalt + FreezeAll; a subsequent TryAcquire with cap headroom is still Denied -- proves fail-closed, not "assume zero spend."
3a Freeze pause-not-drain Gate3_FreezeAll_PauseNotDrain_ResumeGrantsNoDoubleSlotNoOrphan A granted slot survives FreezeAll untouched (GrantedSlotCount unchanged); new acquires denied while frozen; Resume changes stage back to Running without altering GrantedSlotCount (no double-slot); the original slot still releases cleanly afterward (no orphan); cap is fully usable again post-resume.
3b Kill is terminal Gate3_Kill_IsTerminal_ResumeThrows Kill sets FreezeState.Killed = true; Resume after Kill throws InvalidOperationException.
4a Concurrency correctness Gate4_ParallelAcquires_NeverExceedCap 200 real parallel Task.Run acquires at tier RED (cap 2) against a fixed virtual instant: exactly 2 granted, GrantedSlotCount == 2.
4b Concurrency correctness under churn Gate4_ParallelAcquireRelease_NeverExceedCap_UnderChurn 500 parallel acquire+release cycles at cap 2: high-water mark of concurrently-granted slots never exceeds 2.
5 Audit ordering/immutability Gate5_Audit_SpawnDenyTripFreeze_OrderedImmutableChain_NoGapDuringTrip Tier BLACKOUT (cap 0) forces immediate deferrals; sequence numbers are gap-free and strictly increasing; every PriorEventId resolves inside the log (reconstructible offline); Deny, Trip, and Freeze are all present with no silent transition; per-subject chain (hostile-forkbomb) links each event to the immediately-prior event for that same subject.

Full suite: dotnet build -warnaserror -> 0 warnings, 0 errors. dotnet test -> 522 passed, 0 failed, 0 skipped (58 of those under the Autonomy namespace: 50 pre-existing FLT-133 contract tests + 8 new FLT-113/132 tests). No existing test was modified.

Proof nothing is wired into the running governor

grep -rln "UsageGovernor.Core.Autonomy\|AutonomyBreaker\|IAutonomyBreaker\|HostileSpawner" --include="*.cs" . \
  | grep -v "^./src/UsageGovernor.Core/Autonomy/" \
  | grep -v "^./tests/UsageGovernor.Tests/Autonomy/"
# (empty)

Confirmed empty against src/UsageGovernor.Cli/Program.cs (the CLI dispatch table), ScanService.cs (the scan/tier/state pipeline), and every AgyRaft*.cs file (the Raft admission authority). git status --short on this branch shows only two new, untracked directories -- no existing file was touched:

?? src/UsageGovernor.Core/Autonomy/
?? tests/UsageGovernor.Tests/Autonomy/

HostileSpawner.TestOnlyHardOff = true is an explicit, grep-able statement of the hard-off, on top of the structural fact that the type only exists inside the test assembly and is never referenced by anything under src/.

What FLT-115 (not built) will later call

IAutonomyBreaker (src/UsageGovernor.Core/Autonomy/IAutonomyBreaker.cs) is the intended call surface: TryAcquire before spawning, Release when a spawned actor finishes, RecordSpend as it burns tokens/dollars, Heartbeat on a cadence shorter than its declared MaxAgeSeconds, and CurrentFreezeState/AuditTrail for observability. FLT-115 would also need to supply AutonomyBreaker.SetTier calls driven by the real TierPolicy.Compute output (currently the breaker's tier is caller-set and defaults to "GREEN" -- there is no live wiring to ScanService's tier today, by design).

TODOs / explicitly deferred (not this ticket)

  • FLT-114 (freeze enforcement): IFreezeCommandSink is a marker-shaped signal only. Nothing in this repo pauses or kills a real running actor -- that requires a real process/session registry FLT-114 owns. Full TaintDemotionRule.Table enforcement (clamp-to-safe, cannot-cross-floor, transitive propagation on an actual child spawn) is also FLT-114's, not FLT-113's -- the breaker here only enforces the SpawnBlocked column because that's the one column that gates its own concurrency surface.
  • FLT-119 (observability): AuditTrail/CurrentFreezeState are in-memory only; nothing persists or streams them yet.
  • FLT-115 (auto-spawn engine): not built, not callable, per the ticket's explicit scope.
  • Concurrency-storm tunables (ConcurrencyStormWindowSeconds, ConcurrencyStormDeferralThreshold) are FLT-113-owned defaults (5s / 10 deferrals), not council-frozen numbers -- revisit once real fork-bomb telemetry exists.
  • AutonomyBreaker.SetTier has no live caller yet; a future wiring task (not this ticket) would connect it to TierPolicy.Compute's output once a decision is made about whether the breaker should react to governor-tier changes automatically or only on an explicit push.
  • Persistence: the audit log and all breaker state are process-memory only (matches "pure/testable, not wired into the running governor" -- a persistent audit store is part of a future wiring ticket, not this skeleton).