Skip to content

DESIGN — Vendor usage-reset feed watcher (FLT-26) — v2, COUNCIL-APPROVED 2026-07-17

Council: llm-council fable tier (Claude/Codex/Gemini/Grok/local + chairman synthesis). Report: C:\Users\fives\source\repos\llm-council\reports\2026-07-17_152636_spend-control-design-review-usage-governor-flt-2.md Verdict: direction correct (Api-only auto-apply, candidate/confirm split, append-only audit); draft NOT shippable — debounce-unreachable bug, baseline-default overspend landmine, and a falsified lock assumption. All fixes below are incorporated.

Problem

Vendors reset usage limits mid-week out of band. The governor only knows its local burn ledger, so after a limit-hit ratchet-down (BudgetWeighted := floor(burn × 0.8)) or during projection-driven RED it stays conservative when the vendor has restored headroom. Need the inverse path: detect a REAL vendor reset and ratchet the budget back UP — without ever loosening on a false signal. Verified live 2026-07-17: vendor 5h=26%/weekly=37% while local governor RED.

Non-goals

  • Not the CLI balancer (#18), not the fleet admit ledger (#17 — live authority untouched).
  • Never lowers budgets (that stays with the existing limit-hit ratchet).
  • Never edits tier directly — it adjusts config/window inputs; scan recomputes tier. Restore ≠ GREEN: after any restore, force an immediate rescan; if projection still RED, RED stands.

Architecture (C#, UsageGovernor.Core)

1. Normalized observation model

QuotaObservation { Vendor, Window (5h|weekly), UsedPct, ResetsAt?, VendorTimestamp, ObservedAt, Source, Grade, AccountId? } Grade ∈ { Api (documented vendor quota surface or vendor-written session telemetry), PrivateApi (vendor-written but product-internal/undocumented — auto-capable with mandatory corroboration), Status (status page / RSS — NEVER auto-applies), Local (our ledger — never a reset source) }.

Observations + every decision (candidate, verified, applied, skipped+reason) appended to reset-feed.jsonl (same append-only pattern as burn-history.jsonl).

2. Per-vendor probes (authoritative source ranking)

Vendor Primary Grade Fallback (Status, never auto)
OpenAI/Codex codex app-server account/rateLimits/read (documented usedPercent/windowDurationMins/resetsAt) when available; else ~/.codex/sessions/*.jsonl rate_limits (ordered by in-record timestamp, NOT file mtime; file identity hashed so rotation can't fake a drop) Api status.openai.com RSS
Anthropic/Claude OAuth /api/oauth/usage (verified live 2026-07-17: five_hour.utilization, seven_day.utilization, per-limit {kind, percent, severity, resets_at}; auth via existing local Claude Code credentials, identity bound to observation) PrivateApi — strict schema pin; any unexpected shape → parse error → no observation; 401/403/HTML/rate-limit body → fail closed status.anthropic.com RSS
Google/Antigravity none (agy ledger is OUR burn) — Google AI status — candidate-only, permanently, until a vendor-written quota surface exists
xAI/Grok none (untracked plan) — xAI status — candidate-only, permanently

Rules: - Only Grade=Api / PrivateApi can trigger automatic ratchet-up. - PrivateApi (Anthropic) auto-apply requires corroboration jointly: drop test AND resets_at rolled over vs prediction. Either absent → candidate-only. A bare usedPct = 0 with no valid advancing resets_at never auto-applies. - Status signals only produce a reset-candidate audit event + status output line for human confirm-reset. RSS "resolved"/incident wording NEVER promotes to Api grade.

3. Reset detection (real reset vs noise/flapping)

Per (vendor, window, account), comparing consecutive Api/PrivateApi observations:

  • Qualifying observation (#1): all of —
  • Drop test: prevUsedPct − curUsedPct > MaxNaturalDecay(elapsed, window) × 2 + 10pp, MaxNaturalDecay = 100 × elapsed/windowLength.
  • Floor test: curUsedPct ≤ 5 for the auto path (≤ 10 logs candidate-only).
  • Meaningfulness: prevUsedPct ≥ 30.
  • Confirmation (#2) (fixes the council-flagged unreachable-debounce bug — #2 is NOT re-tested against the drop test): an observation ≥ 10 min after #1 showing usedPct sustained ≤ 5 with no intervening rebound. Flap guard: any observation > 10 between #1 and #2 discards the candidate.
  • Same window required on both observations; ObservedAt monotonic; clock skew vs vendor timestamp > 5 min → reject; relative reset fields (resets_in_seconds) anchored to the observation's own vendor timestamp, never local wall clock.
  • Staleness: observations older than 6h ignored (mirrors CodexLimitReader.ObservationFreshness).
  • Action rate limit: max one auto-ratchet-up per (vendor, window, account) per 24h; repeats log candidates only.
  • Weekly reset NEVER touches 5h state; 5h reset never touches weekly state.

4. Ratchet-up (inverse of RecordLimitHit)

ResetRecalibrator.ApplyReset(vendor, window, evidence, config, paths, now):

  • Config gains budgetWeightedBaseline — default null (council: a 1e9 default can loosen a deliberately-lowered manual budget). The limit-hit ratchet-down records the pre-ratchet value into baseline only if baseline is null (first ratchet in an epoch). No baseline → no auto-restore; candidate-only.
  • Baseline lifecycle: expires at the weekly boundary; cleared on manual config edit (detected via stored hash of budgetWeighted at ratchet time), plan/credential change.
  • Auto path (verified event): STEPWISE — BudgetWeighted := min(baseline, floor(BudgetWeighted / 0.8)) per event; with the 24h rate limit this bounds loosening to ~25%/day, symmetric with the down-ratchet.
  • Human path (confirm-reset): full restore to baseline. Human judgment absorbs the false-positive risk.
  • Locking: BOTH paths under GovernorConfig.WithFileLock — including the existing ScanService.Run recordLimitHit branch, which currently has no lock (council finding — fix it in this change). Under the lock, ratchet-up re-reads burn-history and aborts if any limit-hit event is newer than the oldest observation in its evidence (a 429 landing between confirmation and apply must win).
  • 5h window on verified Anthropic 5h reset: sync, don't zero — burn := vendorUtilization × budget, WindowStart derived from vendor resets_at; applied only when vendor 5h util ≤ 5 on BOTH confirmations, and never on a weekly-only reset.
  • Appends reset-restore event to burn-history.jsonl — excluded from ComputeRate math (only scan events feed the rate; verified: ComputeRate already filters on event=="scan").
  • v1 scope: Claude BudgetWeighted only. Codex tier self-corrects each scan; agy pool budgets stay down until human confirm-reset (no Api-grade Google source).

5. Shadow mode (ship default)

Config resetFeed.autoApply — default false. Detector runs fully, logs would-have-applied events to reset-feed.jsonl; human compares against reality for 1–2 weeks, then flips on. Candidates + confirm-reset work regardless.

6. Integration + cadence

  • Local Codex probe piggybacks the existing scan pass (schtask UsageGovernorScan, every 5 min) — no new scheduler.
  • HTTP probes (Anthropic OAuth; status RSS): CLI verb watch-resets, detached schedule 15 min ± 2 jitter, ETag/If-Modified-Since on RSS, hard 3s timeout (10s for OAuth), never blocks scan. Poll only while a ratchet-down or projection-RED is active — nothing to restore otherwise. On HTTP 429/403 from the OAuth endpoint, suspend polling 1–2h. HTTP failure = no observation = no action.
  • New verbs: watch-resets (one poll pass), confirm-reset (requires explicit vendor + window + candidate evidence ID; refuses if no pending candidate or evidence > 6h stale), status extended to show pending candidates + shadow-mode would-haves.

7. Fail-conservative invariants (hard rules)

  1. Only Api/PrivateApi evidence auto-applies; PrivateApi needs joint corroboration; status words never auto-apply.
  2. Ratchet-up restores at most to recorded baseline; null baseline → no auto-restore.
  3. Two-observation debounce (qualify + sustained-low confirm); 24h action limit per (vendor, window, account).
  4. Any parse error / staleness / skew / missing data / schema surprise → no observation → no action.
  5. Every decision audited in reset-feed.jsonl.
  6. Never touches: tier field directly, admit ledger, agy live authority, rate math.
  7. Shadow mode default-off for auto-apply.
  8. Restore ≠ GREEN — rescan decides.

8. Status/RSS fallback probes (v1.1 — implements the §2 fallback column)

Formerly a v1 deferral; now implemented as StatusPageProbe (Core, pure parser) + a status-feed pass inside watch-resets (CLI, HTTP). Design deltas vs the v2 sketch:

  • Feed URLs are config (resetFeed.statusFeeds, vendor+url list; defaults for status.anthropic.com, status.openai.com, xAI, Google Cloud Atom). A vendor moving their status page is a config edit; a wrong/unreachable URL degrades to "no observation".
  • Parser invariants: RSS 2.0 + Atom, item must carry a limit keyword (usage/quota/limit/rate-limit/capacity/reset) AND a publish date within the 6h staleness horizon. Grade = Status by construction — the parser has no code path to any other grade, so "resolved"/incident wording can never promote to Api (§2 rule enforced structurally). VendorTimestamp := ObservedAt (a status item is not a quota reading; incident publish time is used only for the recency gate, otherwise every real incident would trip the detector's 5-min clock-skew rejection). Window := "5h" is a labelling convention to pass feed validation; nothing numeric on the Status path is ever consumed. Per-incident dedupe by feed guid against 24h of reset-feed history AND within the poll pass (title edits and double-configured feeds do not re-candidate); eligible items sorted newest-first before the max-5-per-pass cap (an oldest-first feed must not starve newer incidents behind cached validators). Feed URLs preflighted: https-only, host must not resolve to loopback/private/link-local/CGNAT (misconfig/SSRF defense; DNS-rebinding gap accepted for an advisory-only probe); response bodies capped at 512K chars, overflow = no observation and no validator caching.
  • Detector quarantine (behavior change): Grade=Status observations are routed to their advisory candidate BEFORE the pending-candidate machinery. Previously a same-key status observation would (a) terminally skip an open Api/PrivateApi candidate (RSS noise vetoing real evidence — openai status items and Codex session observations share a null AccountId key) and (b) shadow it out of FindPendingEvent as the newest candidate, orphaning it mid-debounce. Now a status item can only ever open its own status-candidate-only candidate; it can neither confirm, flap-discard, suppress, nor shadow quota evidence. The Grade guard inside the confirmation path remains for Local/future grades, and ResetRecalibrator's auto-path grade guard is untouched.
  • Advisory-only, including for humans: a status candidate surfaces in status output (with incident id + title in source=) as a prompt to investigate. confirm-reset FINDS it, but ResetRecalibrator's evidence floor (≥ 2 observations) refuses a lone status item even with humanConfirmed — restoring on a public RSS sentence alone is out of scope for v1.1; the human confirms against a real quota-grade candidate (for Anthropic the OAuth probe supplies one). Google/xAI status candidates additionally refuse with vendor-not-in-scope (§4 v1 scope) — they are visibility, not actionability.
  • HTTP discipline: hard 3s timeout (vs 10s OAuth), ETag/If-Modified-Since conditional requests persisted in status-probe-cache.json (best-effort; cache loss = one full fetch), 304 → silent, non-2xx/network error → skipped audit event per vendor per pass. Same poll-only-while-constrained gate as the quota probes.

9. Test matrix (council-mandated)

Flap (drop-then-rebound), 7h-stale observation, baseline-unset, concurrent down+up ratchet (limit-hit newer than evidence → abort), OAuth schema break / HTML body / hardcoded zeros without advancing resets_at, DST/clock jump, manual budget edit clears baseline, post-reset local burns preserved by sync, weekly-only reset leaves 5h untouched, debounce actually reachable (regression for the v1 bug), shadow mode logs but does not apply, Grade=Status observation fed to the auto path is rejected by an explicit code guard (house council: rule #1 must be enforced in code, not convention).

§8 additions: status obs on a same-key pending Api candidate neither verifies, kills, nor shadows it (regression pair for the quarantine); lone status candidate is findable by confirm-reset but refused by the recalibrator even human-confirmed; "resolved" wording stays Grade=Status; stale/undated/future items dropped; guid dedupe survives title edits; malformed XML harmless; per-pass incident cap.