BYOK LLM harness (technical)¶

WorkWingman used to hardwire the Claude Code CLI at every place it needed a language model (document tailoring, thank-you notes, résumé extraction, study-content fact-checking, the apply-time page advisor). That coupling is now generalized into one provider-agnostic, bring-your-own-key (BYOK) harness so the user can point the whole app at whichever AI they already have — Claude, Codex, Gemini (Antigravity agy), Grok, Meta/Llama, or any model served by Ollama or an arbitrary CLI, plus BYO-key cloud APIs (Anthropic, OpenAI) — with no code change for new CLI providers.

The fleet is auto-detected: every launch, each registered harness is cheaply probed for whether it's installed, what version it is, and whether it's actually ready to use (logged in / model pulled / key present), and the Connections screen's provider switcher lets the user set the default from that live list. Detection and switching are covered in full below.

Plain-language version: ../plain/llm-harness.md

Out of the box the default provider is claude, so behavior is identical to before for existing users. Nothing here is ours: every harness runs on the user's machine under their login or their key. The app never provisions an account, holds a credential, or hardcodes a key.

The frozen contract¶

Everything above the seam codes against one interface (ILlmHarness):

public interface ILlmHarness {
    string Id { get; }               // "claude","codex","agy","grok","ollama:qwen3:8b","anthropic","openai"
    string DisplayName { get; }
    Task<LlmAvailability> CheckAsync(CancellationToken ct = default);
    Task<string?> GenerateAsync(LlmRequest req, CancellationToken ct = default); // text, or null on ANY failure
}
public sealed record LlmRequest(string Prompt, string? System = null, bool JsonOutput = false);
public sealed record LlmAvailability(
    bool Available, string? Detail = null, string? Account = null,
    string? Version = null, bool Installed = false);
public interface ILlmHarnessSelector { ILlmHarness Default { get; } ILlmHarness? ById(string id); IReadOnlyList<ILlmHarness> All { get; } }

// OPTIONAL, purely-additive UI metadata for the provider switcher — a separate interface so the
// generate/check contract above stays frozen. A harness that skips it falls back to id-based defaults.
public interface ILlmHarnessMeta {
    string Kind { get; }        // "cli" | "local" (Ollama) | "key" (BYO-key cloud)
    string? InstallUrl { get; } // the CLI's install page or the key provider's signup/console page
}

Best-effort contract. GenerateAsync returns the model's text, or null on ANY failure (missing binary, non-zero exit, timeout, unparseable/empty output, transport error). It never throws for an ordinary failure — only a genuine cancellation propagates. This mirrors the best-effort/fallback contract the Claude-CLI services already relied on, so every caller keeps its deterministic offline fallback and simply treats null as "model unavailable."

Detection contract. CheckAsync is a separate, cheap probe from GenerateAsync — it never sends a real prompt. LlmAvailability carries enough for the tri-state switcher:

Available — usable right now (ready).
Installed — present on the machine/reachable but not yet usable (CLI installed but not logged in, Ollama reachable but the model isn't pulled, a cloud key not pasted yet). This is what separates "installed" from "missing": a harness that is neither Available nor Installed is genuinely absent and gets an install/get-key deep-link instead of a "finish setup" hint.
Version — the cheaply-known version string, when one exists (CLI --version, or the configured model name for a cloud-key/Ollama harness so the UI shows which model a key/pull will drive).
Detail / Account — a short human-readable reason and, when known, the logged-in account label. Never a key or secret.

Three families of adapters cover every provider¶

1. `CliLlmHarness` — one config-driven adapter over `IProcessRunner`¶

CliLlmHarness is a single class parameterized entirely by a CliHarnessPreset row. There is no per-provider subclass. A preset carries the executable, the argument template (with {prompt}/{model} placeholders), how the prompt is fed (Stdin / StdinClosed / Argument), the availability probe, and the timeout. The built-in table:

id	launch shape (Windows)	prompt via	probe
`claude`	`cmd.exe /c claude -p --tools ""`	stdin	`where claude` + `~/.claude.json` (logged-in email)
`codex`	`codex exec --skip-git-repo-check {prompt}`	argument	`where codex`
`agy`	`%LOCALAPPDATA%\agy\bin\agy.exe --model "gemini-2.5-flash" -p {prompt}`	argument (stdin not redirected)	`where agy` / file exists
`grok`	`grok -p`	stdin	`where grok`
`meta`	`llama -p`	stdin	`where llama`

Two provider quirks are baked into the presets from live verification:

claude — --tools "", never --allowedTools "". --tools "" is the documented disable ALL tools form. --allowedTools "" is only an allow-list and was proven ineffective against claude.exe 2.1.201 (the Read tool still ran and read back a secret file). Because the claude prompt rides on stdin, an untrusted prompt (a prompt-injected job description, résumé, or interview fact) never touches the command line and can't reach any tool. This security detail is now a preset property, not scattered across three services.
agy — prompt as the value of -p, stdin closed. agy does not read the prompt from stdin (verified live: -p "..." returns text, stdin does not); it hangs if left waiting on interactive input, so stdin is left unredirected (effectively closed in a non-interactive host).

Unknown CLIs are pure config. A brand-new CLI is added with an appsettings entry under WorkWingman:Llm:Cli:<id> (executable + arg template); an existing preset's fields can be overridden the same way. No recompile.

Detection mechanics: CheckAsync. For a CLI preset, CheckAsync runs (in order): (1) is ProbeBinary on PATH (via where/which, ArgumentList-based so a value with a space can't be reinterpreted as two arguments) — falling back to a file-exists check for an absolute path like agy's %LOCALAPPDATA% executable; (2) if present, a cached {binary} --version probe (below); (3) if the preset carries a LoginStateRelativePath (claude's ~/.claude.json), read the account email from it, or treat the existence of LoginStateFallbackRelativePath (~/.claude/.credentials.json) alone as "logged in, email unknown"; otherwise report installed-but-not-logged-in. A preset with no login-state path treats "on PATH" as fully usable (most CLIs carry their own session).

Version cache. A --version probe is a whole extra subprocess, so CliLlmHarness caches the probe task in a static ConcurrentDictionary<string, Task<string?>> keyed by the probe binary (shared by concurrent CheckAsync calls, and shared across two ids that probe the same binary). A null result (missing/quiet CLI) is cached too, so a provider that doesn't answer --version isn't re-probed on every Connections-panel load. The probe itself runs detached from the caller's CancellationToken (CancellationToken.None) so a canceled panel load can't poison the shared cache entry for every later caller; the version text may land on stdout or stderr depending on the CLI, so both are parsed.

2. `OllamaLlmHarness` — local HTTP¶

OllamaLlmHarness POSTs {base}/api/generate with stream:false (default http://localhost:11434, model configurable) and reads the response field. Availability is a cheap /api/tags probe ("server up and is my model pulled?") — reachable-but-not-pulled reports Installed: true, Available: false so the switcher shows "installed" (finish setup: pull the model) rather than "missing". One harness is registered per configured model, id ollama:<model> — so qwen3:8b and gemma4 coexist as distinct selectable providers. Kind => "local", InstallUrl points at ollama.com/download.

3. `AnthropicLlmHarness` / `OpenAiLlmHarness` — BYO-key cloud APIs¶

The "cloud key, no CLI" branch. AnthropicLlmHarness (id anthropic, POST /v1/messages) and OpenAiLlmHarness (id openai, POST /v1/chat/completions) are always registered — so they always appear in the switcher — but report unavailable until the user pastes a key. Both:

Read their key through IApiKeyProvider.Get(providerId, ct), which resolves vault-first (the key pasted on the Integrations screen) then falls back to IConfiguration/env var for a developer's local key — the harness never holds, logs, or returns the key itself; CheckAsync's Detail says "Add a … API key in Integrations…", never the key.
Report Kind => "key" and an InstallUrl pointing at the vendor's own key console (console.anthropic.com/settings/keys, platform.openai.com/api-keys) — a get-a-key deep-link, not an install link.
Carry the configured model name in LlmAvailability.Version even before a key exists, so the switcher can show "which model this provider will use" up front (claude-sonnet-5 / gpt-4o by default, overridable via Anthropic:Model / OpenAi:Model).
Use ResiliencePipelines.NoRetry for the generate call — a paid, non-idempotent POST must not be retried (double-billing risk) — and collapse any transport/timeout/HTTP failure to null, per the shared best-effort contract.

Selection + DI¶

LlmHarnessSelector holds every registered harness and resolves Default from a runtime-switchable preference (LlmProviderPreference), seeded from WorkWingman:Llm:Provider (default claude) and persisted local-first under the store key llm-provider. If the chosen id isn't registered it falls back to the first harness; if none are registered it falls back to a never-available NoOpHarness, so Default is always non-null and callers never NRE.

Registration lives in the composition root: LlmHarnessRegistration.AddLlmHarnesses, which registers the built-in CLI presets, one OllamaLlmHarness per configured model, and — always, unconditionally — one AnthropicLlmHarness and one OpenAiLlmHarness (they just report unavailable until a key exists), so the full provider fleet is visible to the switcher from a clean install with zero config.

Config shape (all optional; defaults keep out-of-box behavior identical)¶

"WorkWingman": {
  "Llm": {
    "Provider": "claude",                     // the DEFAULT harness id
    "Ollama": { "BaseUrl": "http://localhost:11434", "Models": [ "qwen3:8b", "gemma4" ] },
    "Cli": {                                    // add/override CLI presets — UNKNOWN CLIs = pure config
      "grok":  { "Model": "grok-2" },
      "myllm": { "DisplayName": "My LLM", "WindowsExecutable": "myllm", "WindowsArgsTemplate": "-p",
                 "UnixExecutable": "myllm", "UnixArgsTemplate": "-p", "ProbeBinary": "myllm" }
    },
    "Anthropic": { "Model": "claude-sonnet-5" },  // optional override; key itself comes from IApiKeyProvider
    "OpenAi": { "Model": "gpt-4o" }               // optional override; key itself comes from IApiKeyProvider
  },
  "StudyContentVerification": { "UseCodex": true, "Provider": "codex" }  // fact-check model (any harness id)
}

What got rewired onto the seam¶

Caller	Was	Now
`ClaudeDrafter`	hardwired `claude` CLI	`ILlmHarnessSelector.Default` — preserves `ParseFirstJsonObject`, the `DraftHonestyGuard` backstop, the `<<<UNTRUSTED_JOB_POST>>>` fence, and the `TemplateDrafter` fallback
`ClaudeResumeExtractor`	hardwired `claude` CLI	selected harness; empty-result fallback unchanged
`ClaudeThankYouGenerator`	hardwired `claude` CLI	selected harness; deterministic `ThankYouDraftGenerator` fallback unchanged
`CodexStudyContentVerifier`	`codex exec` only	generalized to `LlmStudyContentVerifier` over any harness; the deterministic-first `CompositeStudyContentVerifier` + config gate are kept (default upgrade provider still `codex`)
`StubCodexAdvisorTransport` (threw `NotImplementedException`)	placeholder	replaced by the real `LlmAdvisorTransport` — routes the page-advisor prompt through the selected harness; a null result surfaces as an exception so the "advisor unavailable → pause for user" contract holds

The Claude services still take the same JSON prompts and parse them the same way; only how the prompt physically reaches a model moved into the harness. With Provider=claude the process invocation is byte-for-byte what it was (the rewired tests still assert -p --tools "" on stdin).

Detection + selection flow¶

flowchart TB
    UI["Connections screen<br/>'AI model (bring your own)' card"]
    GET["GET /api/llm/providers/status<br/>(alias of /api/llm/harnesses)<br/>[RequireLocalToken]"]
    Selector["ILlmHarnessSelector.All<br/>every registered harness"]
    Probe["ProbeToViewAsync (per harness)<br/>CheckAsync — never throws, caught and<br/>degraded to Available=false on any failure"]
    Cli["CliLlmHarness<br/>where/which + cached --version +<br/>login-state file"]
    Ollama["OllamaLlmHarness<br/>/api/tags reachable + model pulled?"]
    Key["Anthropic/OpenAiLlmHarness<br/>IApiKeyProvider.Get (vault-first)"]
    Tri{"Available? / Installed?"}
    Ready["status = ready"]
    Installed["status = installed<br/>(needs login / pull / key)"]
    Missing["status = missing<br/>(offer InstallUrl deep-link)"]
    View["LlmHarnessView<br/>id, displayName, kind, version,<br/>status, installUrl, isDefault"]
    Choose["User clicks 'Set as default'"]
    PUT["PUT /api/llm/default { providerId }<br/>[RequireLocalToken]"]
    Pref["LlmProviderPreference.SetAsync<br/>persisted → LocalJsonStore key 'llm-provider'"]

    UI --> GET --> Selector --> Probe
    Probe --> Cli & Ollama & Key
    Cli & Ollama & Key --> Tri
    Tri -->|available| Ready
    Tri -->|installed, not available| Installed
    Tri -->|neither| Missing
    Ready & Installed & Missing --> View --> UI
    UI --> Choose --> PUT --> Pref
    Pref -->|ILlmHarnessSelector.Default resolves this id next| Selector

Connections surface¶

LlmController exposes:

GET /api/llm/harnesses — every harness with a live CheckAsync probe (available, detail, account, version, status, installUrl, kind) and which one is isDefault.
GET /api/llm/providers/status — detection alias of the endpoint above, named for the "which AI providers are installed / ready / missing on this machine" use case; identical payload and gating.
PUT /api/llm/default (local-token-gated) — choose the default provider; effective immediately, persisted. Validates the id against selector.ById first (404 if unknown) before persisting.

Both GETs are gated with [RequireLocalToken]: the payload leaks provider availability plus each harness's Account (e.g. a logged-in CLI's account email) over the API's AllowAnyOrigin CORS policy, so it must not be readable by an arbitrary origin.

ProbeToViewAsync (private, in LlmController) is where the tri-state is computed and where a harness's optional ILlmHarnessMeta is consumed: Kind defaults to "local" for an ollama:-prefixed id or "cli" otherwise when the harness doesn't implement ILlmHarnessMeta; status is "ready" when Available, else "installed" when Installed, else "missing". A probe that throws anything other than cancellation is caught and degraded to Available=false, Detail="probe failed" so one broken provider can never take the whole list down.

The Angular Connections screen renders an "AI model (bring your own)" card: one row per harness with a kind chip (CLI / Local / Cloud key), a tri-state status chip (ready = accent, installed = warn, missing = muted; a "key" provider that's missing reads "No key" instead of "Missing"), the version (when known), detail/account text, an install-or-get-key deep-link button when not ready (installLabel picks "Install CLI" / "Install Ollama" / "Get API key" from kind), and a Set as default button — plus loading / error / empty states (connections.ts / connections.html). The deep-link always opens the vendor's own page in the system browser (window.open) — the app never installs a CLI or provisions a key itself.

Verified live¶

Against the real, locally-installed tools on this machine, each adapter was driven end-to-end through its actual code path (opt-in smoke: WW_LLM_SMOKE=1 dotnet test --filter FullyQualifiedName~LiveHarnessSmoke, in LiveHarnessSmokeTests):

harness	probe	generate
`claude`	logged in (`andrewjonesdev1@gmail.com`)	returned text ✓
`codex`	installed	returned text ✓
`agy` (Gemini)	installed	returned text ✓ (after fixing the prompt-as-argument form)
`ollama:qwen3:8b`	model pulled	returned text ✓

Tests¶

Mocked IProcessRunner per CLI preset (claude/codex/agy launch shapes + best-effort/cancellation, --version probe + cache, login-state file variants) in Llm/CliLlmHarnessTests.cs / CliLlmHarnessAvailabilityTests.cs / CliLlmHarnessQuoteArgTests.cs; Flurl HttpTest for OllamaLlmHarnessTests.cs (generate/tags/error paths) and the cloud-key harnesses (AnthropicLlmHarnessTests.cs / OpenAiLlmHarnessTests.cs — no-key-unavailable, key-present-generate, transport/timeout-failure-to-null, IApiKeyProvider vault-first resolution); the selector's default/fallback logic (LlmHarnessSelectorTests.cs); the tri-state controller mapping, both GET routes, and the [RequireLocalToken] gate (LlmControllerTests.cs); the harness-backed verifier; the advisor transport; and the rewired callers (which still assert the security-critical --tools ""-on-stdin invocation). Frontend: the Connections component's harness load / default marking / tri-state chip / kind chip / install-deep-link / switch / error state. Full suite green.