fleet-routine — Architecture

Audience: engineers working on this repo or integrating with it. Accuracy contract: every claim below is labelled BUILT (running code, tests pass — see BUILD-REPORT.md) or DESIGNED (council-approved plan, no code yet). Do not read a DESIGNED section as a description of current behavior.

1. What this repo is

fleet-routine is the C# scheduling plane for cron-scheduled autonomous "routines" across the 6-harness fleet (Claude, Codex, Gemini/agy, Grok, Odysseus, Ollama) on all house PCs. It owns:

• WHEN a routine fires (Windows Task Scheduler trigger, cron-equivalent),
• UNDER WHAT BUDGET (usage-governor check at fire time),
• governance (fail-closed defer semantics, Pavlok escalation policy).

It explicitly does not own HOW: adapter selection and routing are fleet-harness's job (FLT-1, Python/LangGraph, a separate repo). The runner is designed to invoke fleet-harness routine-fire across a process boundary — prompt-file in, JSON out — and never re-implement routing. This process-boundary seam is a deliberate feature: it preserves fleet-harness's "anti-fork" property (one orchestration engine, not two).

This split was a council decision, not an assumption: council opus 2026-07-17 (FLT-34) ranked proposals D/grok > A/claude > B/codex > C/gemini > E/local and settled 5 AGREE / 2 DISSENT on keeping scheduling and orchestration as separate systems. The two dissenting seats (both from the heaviest-weighted local models, gpt-oss:120b and qwen3-next:80b, per docs/BOARD.md) argued for absorbing scheduling into fleet-harness/Python instead; that proposal was rejected. Report: llm-council/reports/2026-07-17_191704_should-the-fleet-harness-flt-1-the-universal.md (not in this repo — see the fleet-wide llm-council project).

Language: C# first per fleet language policy. fleet-harness itself is Python — a stated, approved deviation, because an outage-continuity orchestrator must not depend on a Claude-adjacent toolchain being healthy.

2. What's BUILT today: Gate 0 (FLT-104) — the doctor auth smoke test

Status: In Review (per docs/BOARD.md). Verified independently by Claude (not on the builder's word): dotnet build -warnaserror → 0 warnings, 0 errors; dotnet test → 18 passed, 0 failed. Built by Codex (offloaded under governor RED), 2026-07-17, on branch wt/WT-6022/FLT-104. See BUILD-REPORT.md for the full build record and reviewer notes.

2.1 Why it exists

The #1 pre-build risk for the whole project: every harness CLI authenticates from an interactive user profile (~/.codex/auth.json, Claude subscription creds, agy/Gemini session, Grok session — all under %USERPROFILE%). A scheduled task running as SYSTEM or a different account loads a different profile, the token isn't there, and the routine fails silently, at 2 AM, forever. A credential-manager sidecar does not fix this — it cannot resurrect a browser-issued OAuth token in SYSTEM context. Gate 0 exists to prove, mechanically and non-interactively, that all six harnesses authenticate correctly from the exact context a real scheduled routine will later run in, before a single line of scheduler code (FLT-105+) is written. Full rationale: docs/FLT-104-gate0-auth-smoke-spec.md §1.

2.2 Solution layout (as it exists on disk)

fleet-routine.sln
src/FleetRoutine.Cli/                  exe "fleet-routine", net8.0, C# 12 implicit usings + nullable
    Program.cs                         CLI entry: arg parsing, doctor verb dispatch, exit codes
    WindowsProbeTask.cs                [SupportedOSPlatform("windows")] — WTS COM lifecycle
    ProbeRunner.cs                     runs the 6 probes against a ProbeRequest, writes the envelope
    Probes.cs                          IHarnessProbe + SubprocessProbe/HttpProbe + FailureClassifier
    Canary.cs                          nonce generation, prompt text, ANSI/CRLF-normalized match
    Gate.cs                            pass/fail aggregation + allowlist rules
    Models.cs                          FailureClass enum, HarnessResult/DoctorResult/ProbeRequest records
tests/FleetRoutine.Tests/              xUnit, 18 tests total
    CanaryTests.cs                     nonce match/mismatch/ANSI-strip/case-sensitivity
    GateTests.cs                       6/6 pass, one-failure, allowlist rules (incl. the auth-can't-be-
                                        allowlisted fix)
    FailureClassifierTests.cs          stderr-text -> FailureClass mapping
    ProbeRunnerTests.cs                end-to-end envelope construction
    SerializationTests.cs              JSON schema round-trip
docs/
    FLT-104-gate0-auth-smoke-spec.md   the spec this section summarizes (read it for full detail)
    BOARD.md                           live ticket status
BUILD-REPORT.md                        build/test verification record + owed items

Package dependency: TaskScheduler (Microsoft.Win32.TaskScheduler NuGet, v2.12.2) — the COM wrapper for Windows Task Scheduler. doctor refuses to run at all on non-Windows (OperatingSystem.IsWindows() guard in Program.cs, exit code 2) — this tool has no cross-platform ambition; the fleet's scheduling host is Windows.

2.3 The doctor verbs

fleet-routine doctor --register-probe                              # one-time, interactive
fleet-routine doctor --noninteractive [--harness <name>...] [--json] [--allow-missing <name>...]
fleet-routine doctor --reset                                        # re-register after password/profile change
fleet-routine doctor --probe-payload                                # internal: what the S4U task itself runs

--harness limits the run to a subset (dev convenience only — the actual Gate 0 acceptance criterion requires the full six). AllHarnesses in Program.cs is ["claude", "codex", "gemini", "grok", "odysseus", "ollama"]; unknown names are rejected at parse time. --allow-missing values must also be in the selected set.

2.4 The S4U probe-task lifecycle (WindowsProbeTask.cs)

The probe must run in exactly the context a real scheduled routine will later run in, or it proves nothing. That context:

• Principal: Andrew's interactive user (DOMAIN\user), never SYSTEM, never a service account.
• Logon type: TaskLogonType.S4U — "run whether user is logged on or not," stored-password logon. This is the one setting that determines whether the profile/token store is even reachable.
• RunLevel: TaskRunLevel.Highest — matches the live fleet-dispatch LAN listener config on 4/4 house PCs (fleet-dispatch/repair-fleet-listener.ps1), though that listener itself uses Interactive+logon-trigger, not S4U — doctor intentionally diverges to S4U because that is the logon type real routine tasks (FLT-106) will use. WindowsProbeTask.cs carries a TODO(FLT-104) to keep the account/RunLevel aligned with that listener config as it evolves.

--register-probe (Register(reset: false)) creates one persistent task named fleet-routine-doctor via TaskService.NewTask(): description, Principal.UserId = current domain\user, LogonType = S4U, RunLevel = Highest, AllowDemandStart = true, MultipleInstances = IgnoreNew, ExecutionTimeLimit = 35 minutes. The single ExecAction invokes the current running executable with doctor --probe-payload (resolved via Environment.ProcessPath, with a dotnet <dll> doctor --probe-payload fallback when running un-published under the SDK). Registration passes password: null — the tool never sees, stores, env-passes, or logs the password. Windows' own LSA/DPAPI credential store, scoped to Andrew's account, custodies the S4U secret; that is the entire point of S4U. --reset calls Register(reset: true), which deletes and re-registers the task (for a Windows password change or profile change — see the ErrorLogonFailure case below).

--noninteractive (RunAsync) does not re-register anything. It: 1. Fetches the existing task via service.GetTask(TaskName); throws ProbeTaskException with a printed remediation command if absent. 2. Validate(task) — fail-closed on any drift: refuses if the principal is SYSTEM (S-1-5-18/SYSTEM/*\SYSTEM), refuses if the principal isn't the expected user, refuses if logon type isn't S4U, refuses if RunLevel isn't Highest, refuses if the task's single ExecAction doesn't match the expected payload command byte-for-byte (guards against a swapped or stale action passing the identity checks but running the wrong payload), and refuses if the task's last run failed with 0x8007052E (ErrorLogonFailure — the specific HRESULT for stale S4U credentials after a Windows password change), each with a printed remediation. 3. Generates a fresh nonce (Canary.CreateToken()), writes request.json ({ nonce, harnesses, allowMissing }) to %LOCALAPPDATA%\fleet-routine\doctor\, deletes any stale result.json, and calls task.Run(). 4. Polls task.State every 500ms against a 35-minute deadline; throws on timeout. 5. Reads result.json, deserializes a ProbeEnvelope, verifies the nonce matches (a stale leftover result file cannot match a freshly generated nonce — this is the anti-staleness check at the outer-command level, distinct from the canary-in-output check at the per-harness level), and cross-checks task.LastTaskResult against the envelope's own Overall (a mismatch is itself a ProbeTaskException). 6. Overwrites Principal/LogonType in the returned result with the values read from the validated task definition, not the payload process's self-report — a hand-run --probe-payload (bypassing the task) could otherwise forge those fields and fake S4U evidence. This is why ProbeRunner's own logon-type guess (Environment.UserInteractive ? "INTERACTIVE" : "S4U", in ProbeRunner.cs) is explicitly a fallback that the outer command always supersedes when it has a validated task to read from.

--probe-payload is what actually executes inside the S4U task. It reads request.json, runs ProbeRunner.CreateDefault().RunAsync(request), and writes result.json atomically (write to .tmp, then File.Move(..., overwrite: true)) so a reader never observes a partial file. Exit code is 0 for overall PASS, 1 for FAIL — this is the exit code RunAsync cross-checks against the envelope in step 5 above.

2.5 The six-harness probe contract (Probes.cs)

Every probe implements IHarnessProbe { Name, Class, RunAsync(ProbeContext, CancellationToken) }. ProbeContext carries the per-run nonce, the temp prompt-file path, and a 5-minute per-probe timeout (ProbeRunner.cs: TimeSpan.FromMinutes(5)).

Subprocess class (SubprocessProbe, base for Claude/Codex/Gemini/Grok) — auth = a token in the user profile, so the SYSTEM/S4U risk applies directly:

Harness	Executable	Invocation	Prompt delivery	Known trap this code handles
Claude	claude	-p --tools ""	stdin (prompt file piped in)	keep the probe tool-free so it can't spawn side effects
Codex	codex	exec -	stdin, then stdin closed	codex exec hangs forever on an unclosed stdin — SendPromptOnStdin=true + explicit process.StandardInput.Close() after the copy; TimeoutIsStdinHang=true so a timeout here reports STDIN_HANG_TIMEOUT, not a generic TIMEOUT
Gemini	agy	-p	stdin, then closed	same stdin-hang trap as Codex
Grok	grok	--prompt-file <path>	not stdin — SendPromptOnStdin=false	grok dies above a 32KB argv; the prompt is always short here, but the code path deliberately never risks argv for this harness

Process plumbing shared by all four: UseShellExecute=false, redirected stdin/stdout/stderr, UTF-8 encoding pinned on stdin and stdout/stderr (mismatch would silently corrupt the canary text — "produces false negatives, not errors," per the spec), CreateNoWindow=true. A child that closes its own stdin early (e.g., because it's already failing unauthenticated) would otherwise throw IOException on the write — that specific exception is swallowed so an auth failure surfaces as a proper FAIL result through the normal exit-code/classifier path, not an unhandled crash of the whole payload. Process-not-found / launch failure (Win32Exception, FileNotFoundException) maps to FailureClass.PATH_OR_EXE_NOT_FOUND.

HTTP class (HttpProbe, base for Odysseus/Ollama) — auth = reachability + model presence, not user OAuth, but still run inside the task context so PATH/network differences surface:

Harness	Endpoints	Missing-model handling
Ollama	GET 127.0.0.1:11434/api/tags then POST .../api/generate (stream:false, num_predict:32)	empty models array → FailureClass.MODEL_UNAVAILABLE
Odysseus	GET 127.0.0.1:7000/health, GET .../v1/models, then POST .../v1/chat/completions (max_tokens:32)	empty data array → MODEL_UNAVAILABLE

Both are hard-coded to 127.0.0.1 — a deliberate scope limit (council decision, spec §9.1): Gate 0 proves the local S4U execution context; a remote box's uptime is a reachability fact, not an auth fact, and folding it in would turn a binary go/no-go into a flaky network inventory check. Cross-host reachability is explicitly out of scope until FLT-107. HttpRequestException maps to FailureClass.NETWORK_UNREACHABLE; a timeout maps to FailureClass.TIMEOUT.

The shared HttpClient (ProbeRunner.CreateDefault()) is constructed with Timeout = Timeout.InfiniteTimeSpan — the linked CancellationToken (5-minute per-probe budget) is the sole timeout authority, so a slow local CPU-bound Ollama generation isn't cut off by HttpClient's own default 100-second timeout before the intended budget is spent.

2.6 The per-run nonce canary (Canary.cs)

CreateToken()  // 4 random bytes -> "FLEET_OK_" + 8 lowercase hex chars, e.g. FLEET_OK_7f3a91c0
Prompt(token)  // "Respond with exactly `FLEET_OK_7f3a91c0` and nothing else."
Matches(output, token)  // Normalize(output).Contains(token, Ordinal)   -- case-sensitive
Normalize(output)       // strip ANSI escape codes (regex), normalize CRLF/CR -> LF

Match rule is deliberately contains, not exact-match: exact-match fails in practice because reasoning models prepend thinking text, chat models add markdown/whitespace, and local models sometimes truncate — contains is "the strictest rule all six reliably satisfy" (spec §4). A harness PASSES only if all of: (1) the call completed within its timeout, (2) exit status/HTTP status indicates success, (3) the canary contains the exact current-run nonce, (4) no interactive prompt blocked it, (5) the temp prompt file was read successfully — proving file-based prompt delivery, the channel every future routine will also use.

The nonce is fresh every doctor invocation — this is what makes a stale result.json or a --help banner incapable of faking a PASS: neither can contain a token that didn't exist until this run generated it.

2.7 Gate aggregation and the allowlist rule (Gate.cs)

Gate.Aggregate(results, allowMissing, logonType) rewrites a FAIL to ALLOWLISTED_MISSING only if the harness name is in allowMissing and its FailureClass is one of the three in AllowlistableFailures: PATH_OR_EXE_NOT_FOUND, MODEL_UNAVAILABLE, NETWORK_UNREACHABLE. Overall is PASS iff every result is PASS or ALLOWLISTED_MISSING.

This is intentionally narrow: --allow-missing may excuse a seat that is genuinely absent (binary not installed) or unreachable (local server down / no model pulled), but it can never launder an auth failure (AUTH_TOKEN_MISSING, AUTH_PROMPT_REQUIRED, DPAPI_DECRYPT_FAILED) or a CANARY_MISMATCH — those stay hard FAIL even if named in --allow-missing. This constraint was added by a council review pass after the initial build (source comment in Gate.cs: "council review 2026-07-18"; test: GateTests.Allow_missing_must_not_swallow_auth_or_canary_failures). See docs/technical/making-of.md for the sequence.

DoctorResult also carries Principal (DOMAIN\user, proving not-SYSTEM), LogonType, Pc (Environment.MachineName), and a SchemaVersion (currently 1) for the JSON output consumed by --json and (eventually) the FLT-105 runner's own fire-time precheck.

3. What's DESIGNED, not built

Everything past Gate 0 is architecture-only. See docs/diagrams/routine-lifecycle.md for the sequence diagram and docs/BOARD.md for live ticket status. Summary, in build order (each blocked on the previous):

• FLT-105 — run-now vertical slice. The actual C#↔fleet-harness seam: a thin runner that invokes fleet-harness routine-fire as a subprocess, prompt-file in / JSON out, with no scheduler wiring yet (a manual "run this routine now" path). Blocked by FLT-104.
• FLT-106 — WTS trigger. Real routine tasks registered via the Task Scheduler COM/API, trigger-only semantics: reboot/sleep survival, StartWhenAvailable-style missed-run catch-up. This is a different WTS usage than doctor's single persistent probe task — FLT-106 owns per-routine task lifecycle at scale. Blocked by FLT-105.
• FLT-107 — multi-PC JetStream replay. NATS JetStream transport on the frozen fleet.v1 envelope, plus Claude-native output normalization. Two forked design options noted on the board: subprocess/sync vs. NATS-command/async. Also the home for cross-host reachability checks that Gate 0 explicitly excluded (remote Odysseus/Ollama targets). Blocked by FLT-106.
• FLT-108 — Pavlok dedupe/escalation + full cap enforcement. missedRunPolicy = skip-retry, never flush. Blocked by FLT-107.

Cross-cutting design notes already fixed by the council (not yet exercised by any code):

• Runs on NullLedger for now — FLT-1's own Gate 2 (governor cap-tracking) is still open; do not wire agy into cap_proximity until that lands.
• Signing identity = sessionId, not worktreeId — a routine fire is attributed to the session that scheduled/ran it, not the worktree it happened to execute in.
• Emits on the frozen fleet.v1 bus envelope (see the fleet-wide fleet-bus-envelope-fleetv1 design note) — no new envelope version for this project.
• Governance defer semantics: a governor "defer" at fire time means skip this tick only — never queue-and-flush a missed run later, which would violate the budget the defer was trying to protect.

All five tickets (FLT-104 through FLT-108) anchor to the parent epic FLT-34.

4. See also

• docs/technical/references.md — external references (WTS S4U, HMAC/nonce canaries, the harness CLIs, cron) and links for the patterns used here.
• docs/technical/making-of.md — the design/build journey, beat by beat, from git log.
• docs/diagrams/ — all four diagrams referenced above, Mermaid source.
• docs/plain/OVERVIEW.md — the same material, no code, for a non-engineer reader.