WorkWingman AI-Usage Metrics — METRICS.md

Status: WW-66 (metering + ledger + isolation enforcement), first slice of the TROI epic. Satisfies the §7 hard gate of the Token-ROI Metrics Standard v1.0 — this repo's compliance doc, referenced from docs/technical/byok-efficiency.md §1.

Scope: this document describes what WW-66 actually measures today. It does not define cost model math, savings claims, or routing policy — those are later tickets (WW-67 rate cards/governor, WW-69 router, WW-71 durable-outcome linking). Read docs/technical/byok-efficiency.md for the full nine-part design; this file is the narrower "what's shipped and how it's honest" record.

1. What is metered

Every call through any ILlmHarness (src/WorkWingman.Core/Interfaces/ILlmHarness.cs) — Anthropic, OpenAI, Ollama, and every CLI harness (Claude Code, Codex, agy/Gemini, Grok, Meta) — is metered. There is exactly one door: MeteringLlmHarness (src/WorkWingman.Infrastructure/Llm/MeteringLlmHarness.cs) wraps every registered harness in DI (LlmHarnessRegistration.cs), so a caller cannot get an unmetered handle to a provider by construction — metering isn't each caller's responsibility, it's the registration's.

This meters the user's own BYOK keys / installed CLIs, only for WorkWingman workloads, only while WW runs. It is not a general LLM-telemetry product.

2. The ledger is content-free by construction

AiUsageEvent (src/WorkWingman.Core/Models/AiUsageEvent.cs) is a sealed record with no free-text field of any kind — no prompt, no response, no message body. A reflection test (tests/WorkWingman.Tests/AiUsageEventContentFreeTests.cs) asserts no property name contains "prompt", "response", "content", "text", "message", or "body", and that the only string-typed properties are Provider and Model (both non-secret labels, never the key and never generated text). A resume, a job description, or a model's answer is structurally impossible to record here — there's nowhere to put it.

Schema (v1):

Field	Meaning
Id	Stable id stamped at append
Ts	UTC completion time
Provider	Harness id, e.g. anthropic, openai, ollama:qwen3:8b — identifies the key/CLI slot, not a secret
Model	Model name when the harness/provider surfaces one (empty for CLI harnesses that don't)
TaskClass	Declared workload intent (LlmTaskClass) — required on every call, rejects Unknown at the harness boundary
Status	Ok | Empty | Failed — mirrors the harness's own null-on-failure contract
LatencyMs	Wall-clock latency of the harness call
InputTokens / OutputTokens	See §3
SchemaVersion	Lets later shards add fields (cache tokens, etcUsd, routeDecision, outcomeRef per the TROI design) without breaking readers of older rows

This is a deliberate subset of the full ledger shape sketched in byok-efficiency.md §1 (keyIdHash, feature, cache-token breakdown, et/etcUsd, rate-limit snapshots, routing decisions, outcomeRef) — those fields arrive with WW-67/69/71 once there's a metered baseline to build on.

3. Token capture — real numbers where the provider gives them, honest nulls where it doesn't

Token counts come from an ambient side-channel, AiCallContext (src/WorkWingman.Core/Interfaces/AiCallContext.cs): MeteringLlmHarness opens a scope (AiCallContext.Begin()) around each inner GenerateAsync call; a harness that parses a provider usage block calls AiCallContext.Report(input, output, model) while inside that call; the decorator reads scope.Usage once the call returns and stamps it onto the ledger row. This flows the numbers back without changing the frozen ILlmHarness.GenerateAsync(LlmRequest, CancellationToken) -> Task<string?> signature.

• Anthropic (AnthropicLlmHarness) — reads usage.input_tokens / usage.output_tokens from the Messages API response.
• OpenAI (OpenAiLlmHarness) — reads usage.prompt_tokens / usage.completion_tokens from the Chat Completions response.
• Ollama (OllamaLlmHarness) — reads prompt_eval_count / eval_count from /api/generate.
• CLI harnesses (Claude Code, Codex, agy, Grok, Meta — CliLlmHarness) — report nothing. These drive a subscription via a subprocess; there is no per-call token count to read. Their ledger rows carry InputTokens = null, OutputTokens = null.

Null means "not measured," never "zero." A CLI call that consumed real subscription capacity is recorded with null tokens, not 0 — collapsing "no data" into "zero cost" would understate real usage and is exactly the kind of quiet-zero the standard's honesty requirement rules out. Tokens are reported even when the call's text comes back empty or the provider call otherwise "fails" post-response — the report happens right after the response JSON is parsed, inside the same try block, because tokens were consumed regardless of what the text turned out to contain.

4. Storage: a separate, append-only shard family

AiUsageLedger (src/WorkWingman.Infrastructure/Services/AiUsageLedger.cs) is a LocalJsonStore-backed recorder, one collection per month: ai-usage-{yyyy-MM}. This is a separate shard family from metrics-events-* (the Value Metrics event log) by design — different PII postures, must never share a file. Appends are gated by a semaphore around a load-add-save sequence so concurrent callers (e.g. JobFit's parallel narrative calls) don't clobber each other's rows.

Never throws. RecordAsync swallows any storage failure — a broken ledger write must never take down the feature that made the AI call. This mirrors MetricsEventLog's existing best-effort contract exactly (IAiUsageRecorder, src/WorkWingman.Core/Interfaces/IAiUsageRecorder.cs).

5. Structural coverage guarantee — the hostname gate

A test (tests/WorkWingman.Tests/Architecture/ProviderHostnameGateTests.cs) scans every .cs file under src/ and fails the build if a provider hostname literal (api.anthropic.com, api.openai.com, generativelanguage, api.x.ai) appears outside src/WorkWingman.Infrastructure/Llm/ — the sole door. This is a structural guarantee, not a procedural one: if a future call site hits a provider endpoint directly instead of going through an ILlmHarness, that call bypasses MeteringLlmHarness and would silently escape the ledger. The test catches that the moment it happens rather than relying on a reviewer noticing during code review.

6. What this does NOT claim

Per the standard's "meter before optimize" principle and the WW-66 build-order note in byok-efficiency.md §9: this ticket makes no cost, dollar, or savings claims. There is no etcUsd field, no plan/tier pricing table, and no MEASURED-vs-MODELED savings computation yet — those require a metered baseline window first (WW-67 rate cards + governor, WW-69 router savings, WW-70 compression savings). Everything in this document is a measurement, not a valuation. When cost figures do arrive in later tickets, they will be labeled MEASURED (calls actually served cheap/free with a priced same-taskClass baseline) or MODELED ("a stronger model would have cost X") and the two will never be summed, per the parent standard §5.

7. Test coverage

• tests/WorkWingman.Tests/Llm/MeteringLlmHarnessTests.cs — decorator behavior: one event per call, status mapping, provider/taskClass/latency pass-through, token capture via AiCallContext, cancellation re-throw with nothing recorded, boundary-guard rejection of Unknown task class.
• tests/WorkWingman.Tests/AiUsageLedgerTests.cs — append/read round-trip, monthly shard naming, merge across shards oldest-first, concurrent-append durability, isolation from metrics-events-*, never-throws on a broken store.
• tests/WorkWingman.Tests/AiUsageEventContentFreeTests.cs — reflection proof the ledger row type cannot hold content/PII.
• tests/WorkWingman.Tests/AiCallContextTests.cs — the AsyncLocal report/scope mechanics, including the no-op-when-no-scope case.
• tests/WorkWingman.Tests/Architecture/ProviderHostnameGateTests.cs — the structural hostname gate described in §5.
• tests/WorkWingman.Tests/Llm/{Ollama,Anthropic,OpenAi}LlmHarnessTests.cs — existing harness suites, extended coverage for the new usage-parsing DTO fields is implicit (their HttpTest mocks don't need to include a usage block; AiCallContext.Report(null, null, ...) is harmless when the field is absent).