Skip to content

BYOK AI-Efficiency Stack — design (council-ratified)

Plain-language version: How WorkWingman Spends Your AI

Status: design ratified 2026-07-10 (llm-council fable tier, unanimous consensus; report: llm-council/reports/2026-07-10_124945_design-sitting-workwingman-byok-ai-efficiency-st.md; house gaming-PC council concurred). Build tickets: WW-66..WW-72 (TROI epic).

Parent standard: Token-ROI Metrics Standard v1.0 — this stack is the WorkWingman adaptation. Compliance with that standard's §7 hard gate is required; this repo's METRICS.md will be added in WW-66.

Scope rule: everything here meters and governs the user's own BYOK keys, only for WW workloads, only while WW runs. The models get fenced task prompts and nothing else; the metrics subsystem gets usage numbers and nothing else.

1. Metering (WW-66)

  • Seam: the existing AIH provider-agnostic harness (delivered Jul 7–8, ce6e40a) is the sole LLM door. Metering middleware (TokenMeteringMiddleware) wraps every harness call. No new gateway; the council's ILlmGateway role is played by the AIH harness interface.
  • Structural coverage guarantee (not procedural): Roslyn analyzer / CI grep fails the build on provider hostnames (api.anthropic.com|api.openai.com|generativelanguage|api.x.ai) or provider SDK references outside the harness namespace. Debug builds add a DelegatingHandler asserting every LLM-host request carries a harness correlation-id.
  • Ledger: ai-usage/<yyyy-MM>.events.jsonl, append-only monthly (mirrors Value Metrics event files). Sealed record — no free-text field able to hold prompt content exists in the type (it has strings for model ids and enums, but nowhere to put what you wrote), so resume/PII text cannot leak into telemetry: {ts, provider, model, keyIdHash, taskClass(enum), feature(enum), tokens{input,cacheCreate,cacheRead,output}, et, etcUsd, latencyMs, status, rateLimitSnapshot, routeDecision, outcomeRef(validated id)}.
  • ET/ETC formulas per parent standard §2. Ollama calls go through the same harness + ledger ($0 cash, tokens tracked).

2. Isolation (WW-66)

  • Stateless API calls; tools: [] hard-coded in the API adapters — no function-calling code path compiled into them. This does NOT extend to the CLI harness presets. claude passes --tools "" and grok a deny-all rule, but the codex preset launches with tools ENABLED (codex exec --skip-git-repo-check, no suppression) — WING-431. Until that is fixed, no product-wide tool-disablement claim is true. PromptSafety fencing is applied on the drafting paths, NOT on every prompt — InterviewQuestionCoachService does not apply it today (WING-417 audit). The original "on every prompt" wording overstated coverage; fix the gap or state the scope, but do not restate the universal.
  • Prompt builders select fields explicitly, per call — ClaudeDrafter.BuildProfileBlock and BuildFactPackPromptBlock hand-build labeled text blocks; no code path serializes a domain aggregate into a prompt.
  • There is no redaction step before prompt assembly, deliberately. Drafting tasks send the person's real identity — ClaudeDrafter appends Contact.PreferredName if set, otherwise LegalFirstName + LegalLastName (one or the other, not all three) — because a tailored resume and cover letter cannot be written without it. Email, phone and address are not sent; the app inserts those into the finished document afterwards. Minimization here is field selection and task scoping, not redaction. Do not describe this stack as redacting PII before it reaches a model; it does not, and the feature requires that it does not. (LlmTaskClass exists but drives routing, metering and timeouts — it is not a field allowlist.)
  • What DOES bound the prompt, each with its real scope:
  • PromptSafety fencing of untrusted input, with forged-marker neutralization so a job description cannot close the fence early, plus a no-URL / no-image-tag output rule closing the markdown-exfil vector — drafting paths only; not InterviewQuestionCoachService.
  • Fact-pack grounding, where AcademicFactPack.ToPromptBlock() is presented as "the ONLY facts you may use, cite every claim by these exact ids" — student path only (StudentFactPackInstruction + FactPackCiteGuard). ClaudeDrafter.StandardInstruction requires JSON but does not require citations, so the professional path has no citation enforcement.
  • DraftHonestyGuard, which rejects a draft stating an employer, degree or date the profile does not contain — runs on the professional drafting path, and is the real backstop there.
  • Responses are data, never executed by us — no code path evals or shells model output, and drafters parse it to markdown strings. Output length caps apply. Carve-out: on the codex CLI preset the model's own tool access is live (WING-431), so "the model cannot cause anything to run" is not assertable there until that is fixed — a distinct claim from "we do not execute what it returns", which holds everywhere.
  • Key material: harness resolves keys at call time via the existing vault key-broker. The metrics subsystem reads ledger files only. CORRECTION (WING-417): this was previously described as "enforced by project graph: Metrics project has no reference to the vault project." There is no WorkWingman.Metrics project. AiUsageLedger and the vault (Vault/KeePassVaultService.cs) both live in WorkWingman.Infrastructure, so no assembly boundary separates them. What actually keeps the ledger content-free is the ledger record TYPE having no free-text field able to hold prompt content — a real and checkable property, but type discipline, not project-graph isolation. Do not cite an assembly boundary that does not exist.

3. Plan/tier awareness (WW-67)

  • Declared: at key setup, user picks plan from dropdown (per provider: e.g. Anthropic Free/Pro/Max/API-PAYG; OpenAI Plus/Pro/API tier; Gemini free/AI-Pro). Stored in settings JSON, not the vault (tier isn't secret; vault keeps secrets only). Per key: alias, provider, declaredTier, userMonthlyWWCap, reservePercent (personal-chat headroom WW may never touch), model allowlist.
  • Observed: adapters capture anthropic-ratelimit-* / x-ratelimit-* / 429 retry-after into PlanObservation snapshots; inference compares observed limits vs declared tier's expected.
  • Mismatch surfaces in UI ("headers suggest Pro, you declared Max — fix?") — never silently overwritten. Every plan figure carries a confidence tier: declared | header-confirmed | inferred | unknown.
  • %-of-window = ET consumed in provider window ÷ tier's modeled capacity. Capacity table = editable JSON with source URLs, labeled MODELED.

4. WW usage governor (WW-67)

Thresholds computed against the WW cap = userMonthlyWWCap × (1 − reservePercent) — WW throttles itself before touching the user's personal quota. Per key, per plan window:

Tier Threshold Behavior
GREEN <60% WW cap everything runs
YELLOW 60–80% pause overnight enrichment, study suggestions, council; user-initiated tailoring + core apply continue; prefer local/cheap routes
RED 80–95% user-clicked actions only, single cheapest capable model, banner
BLACKOUT ≥95% or hard 429 all WW AI paused for that key; failover to other keys only if user opted in; user override shows cost warning

Fixed degradation order: enrichment → suggestions → council → tailoring-assist → core apply last.

5. Router (WW-69)

Task classes → models by fit + remaining budget, governor-aware. Defaults: parse/extract/classify/ dedupe → local Ollama else cheapest key; job-fit/ranking → cheap cloud; resume prose/interview answers/cover letters → strongest key; study suggestions → mid. User-visible + overridable (RoutingRulesGrid). Savings: MEASURED = cache reads + calls actually served at $0/cheap with a priced same-taskClass same-window baseline; MODELED = "strong model would have cost X". Never summed (parent §5).

6. WW caveman — compression boundary (WW-70)

Boundary = artifact type, not call site. taskClass (required enum on every harness call) gates it: compression allowed for {parse, extract, classify, dedupe, score, summarize-internal}; artifact-producing classes (resume, coverLetter, thankYou, interviewAnswer, learningPlan) structurally reject the compressed pipeline — user-facing artifacts are always full quality. Savings = MODELED unless later A/B-measured.

7. Metrics tab + durable outcomes (WW-68, WW-71)

Coordination with Value Metrics (docs/technical/value-metrics.md, in flight as of 2026-07-10 — see TROI-COORDINATION-NOTE.md at repo root): the /metrics dashboard belongs to the Value Metrics section; WW-68 ships as a section inside it, not a second page. The AI ledger is a separate shard family (ai-usage-YYYY-MM) reusing the MetricsEventLog append-only/never-throws mechanics — never writing into metrics-events-*. WW-71 consumes Value Metrics events (apply.submitted, application.status.changed, future document.* adoption events) via stable ids for outcomeRef, and reads apps-per-attention-hour from the summary endpoint rather than recomputing it.

Headline: $ per QUALIFIED tailored application submitted and not withdrawn within 7 days — "qualified" = the Value Metrics definition (qualifiedApplicationsSubmitted: submitted AND deterministic skill-coverage Good+, re-denominated by the 2026-07-10 Kerr sitting, commit 2414106). Raw submission counts are spray-gameable; the qualification gate is mechanical (deterministic skill matching), not model-graded, so it passes the self-grading ban. Model-graded quality multipliers (ATS score × count, quality-score-weighted adoption) remain banned — a model grading its own output is the Kerr countable-criteria trap. WW-71 consumes qualifiedApplicationsSubmitted, activeMinutesPerQualifiedSubmission, and responsesPerTenQualified from GET api/metrics/summary — never recomputed. fit.analyzed events also carry adjacentCount (transferable skills, commit 6285670) — available to WW-71 for outcome-correlation analysis from event history; deliberately NOT part of the qualified gate or coverage ratio (separate signal by design, per the Value Metrics session).

Metric Counts when Gaming vector Guardrail
App submitted + kept 7d (headline) user-confirmed submit, not withdrawn spam-apply burn-maxxing response rate + apps-per-attention-hour (Value Metrics activation metric)
Artifact adopted explicit user keep/export/send auto-accept slop, tiny-edit farming adoption = explicit action + min meaningful diff shown
$/outcome — cheap-model-everything (frugality-maxxing) regenerations-per-artifact (rework count)
Interviews earned recruiter screen logged luck-noisy, lagging context line only, never sole KPI
Learning task done study item completed + retained checklist spam must map to target-role skill gap

Tab shows per key: spend this window, %-of-plan (with confidence badge), $/durable-outcome, savings in separate MEASURED / MODELED panes, routing-decision log, governor tier banner. Real wired Angular per house policy — no stubs.

8. WW council (WW-72)

Multi-BYOK-model deliberation (independent answers → peer review → synthesis) for high-stakes artifacts only: resume finals, key interview answers, high-stakes cover letters. Learning plans and project descriptions eligible on user request. Never auto-convened; disabled at YELLOW+.

Consent modal before every run: "Council Mode (3 models). Est. cost: $0.15. ~15s. Window impact: 4% of Anthropic key." — per-seat list with one-click seat drop, max-spend cap, explicit "uses your keys now" label, [Approve] / [Single model instead]. Tracking: $/adopted-council-artifact — adoption MEASURED, uplift vs single-model MODELED.

9. Build order

  1. WW-66 metering + ledger + isolation enforcement (meter before optimize — no savings claims without a baseline window)
  2. WW-67 rate cards + declared tiers + header observation + governor
  3. WW-68 Metrics tab + Keys/Plans UI
  4. WW-69 router + visible overrides
  5. WW-70 compression policy
  6. WW-71 DurableOutcomeLinker
  7. WW-72 council + consent UX + adoption tracking