Token-ROI Metrics Standard (v1.1)¶
Plain-language version: The Token Metrics, In Plain English v1.1 adds §12 (lifecycle-phase spend); v1.0 sections unchanged.
Status: Council-ratified 2026-07-10 (llm-council fable tier, consensus A-D-B-C-E; report: llm-council/reports/2026-07-10_123239_design-sitting-token-roi--an-ai-spend.md)
This document is a HARD GATE. Every repo built by AI agents in this shop MUST carry a METRICS.md conforming to the template in §8, and the gate checks in §7 must pass. The board web UI must render the panel in §9.
1. Why this exists (the folly)¶
Steven Kerr, On the Folly of Rewarding A, While Hoping for B (1975): reward systems pay off for measurable behavior A while hoping for outcome B — A gets done, B doesn't. Root causes: fascination with objective/countable criteria, overemphasis on visible behaviors.
The industry is currently committing this folly with AI tokens, in both directions:
- Burn-maxxing (rewarding spend): "Tokenmaxxing" leaderboards at Amazon, JPMorgan, Meta, Disney — one Disney employee logged 460,000 AI interactions in 9 days gaming a leaderboard. Salesforce engineers asked AI to summarize documents they already understood to hit usage targets. GitClear (Jan 2026): heavy AI users show 9.4× code churn; initial 80–90% acceptance falls to 10–30% after revision accounting. DORA 2025: +21% PR merges per dev, but org throughput down 1.5% per 25pp AI adoption, stability down 7.2%. Token spend is the new lines-of-code — an input measured as if it were an output.
- Frugality-maxxing (rewarding thrift): the same folly inverted. Measured case (Insight, Jul 2026): the cheapest model produced a $17 cost-per-merged-feature vs a stronger model's $7 — weak output triggered extra review cycles that cost more than the model saved. McKinsey: 60% of agentic cost is refinement/rework, not first generation. Starving quality to save tokens moves cost downstream and multiplies it.
Rule zero: raw token volume — up or down — is never a KPI. Only cost per durable outcome is.
Sources: CIO "Tokenmaxxing", GitClear 2026, DORA 2025 report, McKinsey "Unleashing developer productivity with generative AI", Glean token-efficiency metrics, Insight "Cost to a Merged Feature".
2. Cost unit: Effective Token Cost (ETC)¶
All spend is normalized to dollars, never abstract token counts (abstract units hide model-price deltas):
ET = 1.0·input + 1.0·cache_creation + 0.1·cache_read + 4.0·output (per call)
ETC = Σ over calls: ET-weighted tokens × model $/MTok rate card
Output weighted 4× input per real price ratios; cache reads at 0.1×. Rate card lives in pricing.json (per model id, versioned; local Ollama models priced at $0 cash but tracked in tokens + wall-clock for capacity accounting).
Every ledger row carries confidence: exact | measured | estimated | unknown. Claude JSONL usage blocks = exact. Ollama proxy counts = exact. Parsed CLI stdout = measured. chars/4 fallback = estimated.
3. The metric set (ranked)¶
| # | Unit | Metric | Formula | Notes |
|---|---|---|---|---|
| 1 | Ticket (primary KPI) | CPDT — Cost Per Durable Ticket | Σ attributed ETC including ALL rework/review cycles ÷ tickets closed-and-not-reopened-14d | The $17-vs-$7 case falls out automatically: cheap slop reopens, its rework ETC rolls back into the numerator |
| 2 | Commit | $/surviving commit | Survival = ≥70% of added lines live at 14d (git blame), binary | Deliberate refactor commits tagged [refactor] excluded from blame penalty |
| 3 | Feature/epic | $/merged feature | implementation$ + review$ + 30d post-merge-fix$ | |
| 4 | Guardrail (never KPI) | Rework Ratio | ETC after first "done" claim ÷ total ETC | Trend vs McKinsey 60% baseline |
| 5 | Prompt/turn | NO impact KPI — cost gauges only | ETC, trajectory efficiency (ET/tool-step), cache hit % | Turns have no outcomes. Scoring them = Kerr's countable-criteria trap |
| 6 | Council run | $/adopted decision | verdict traceable to commit/ADR within 7d (mechanical check, not self-report) | |
| 7 | Fleet gauges (never rewarded) | %-of-5h window, %-of-week, cache hit %, output/input ratio | from usage-governor state |
Impact numerator (solo shop, zero self-grading): ticket closed + not reopened for 14 days + constituent commits pass blame-survival. Three mechanical signals. Weighted composites, value points, and self-assessed "usefulness" scores are banned (council: tunable-to-flatter-yourself).
4. Attribution (session tokens → commits/tickets/features)¶
Two-tier:
- Primary — explicit:
WW-nn(or repo ticket id) in commit subject orAI-Ticket:trailer; SessionStart hook records sessionId→ticket on branch checkout. - Fallback — deterministic: session JSONL
cwd + gitBranch + timestampjoined to commits on same repo/branch within[session_start − 15m, session_end + 2h].
Concurrent sessions, same branch: nearest-session-by-commit-time. Ambiguous residue goes to a visible unattributed bucket — never silently pro-rated, excluded from headline CPDT until resolved. Unattributed > 20% of ETC = the attribution system is broken; fix the system, distrust the numbers.
Ledger event taxonomy: AiTurn, CouncilSeat, Commit, BoardTransition, GateResult.
5. Savings accounting — two ledgers, NEVER summed¶
Test: if the counterfactual was never priced by an API, it is MODELED, not saved.
MEASURED (real dollars):
- Cache reads: (input_rate − cache_read_rate) × cache_read_tokens.
- Model-ladder routing: (baseline_rate − actual_rate) × tokens, valid only if the work passed gates first try; escalation re-runs count as negative savings. Baseline = the configured default route (sonnet, per escalation-ladder standard) — not the ceiling model; comparing everything to Fable inflates savings.
- Subagent offload: measured only when parent-tier and subagent runs are both priced on the same task class; otherwise report routing-rate stat only.
MODELED (labeled estimate band, separate column, never added to dollar totals):
- Caveman compression est_saved (assumes counterfactual verbosity).
- Governor deferrals: no dollar credit ever (prevented spend is unknowable); log event counts + tier. Empirical-median-of-deferred-class permitted only as a MODELED row with baselineSource: empirical_30d.
Every savings row: method + baselineSource + confidence.
6. Capture (per-AI, per-machine)¶
| Source | Method | Confidence |
|---|---|---|
| Claude Code (all 5 PCs) | parse session JSONL usage blocks (existing, complete) | exact |
| Ollama (local seats, house councils) | reverse proxy on :11434 → real port; persists prompt_eval_count/eval_count. Only exact non-Claude source — build first |
exact |
| Codex / Gemini (agy) / Grok | PATH-shadowing wrapper shims: run real CLI, parse stdout/stderr token lines; fallback chars/4 flagged estimated |
measured/estimated |
| llm-council runs | each seat appends one ledger line before chairman synthesis; missing seat line ⇒ run marked telemetry:incomplete |
per-seat |
Unified ledger: append-only JSONL per machine, {ts, machine, tool, model, in, out, cacheRead, cacheCreate, costUsd, confidence, sessionId, cwd, branch, ticket?}, nightly merge to primary PC. Manual token entry is banned (dies in a week, poisons the ledger).
7. The hard gate (mechanical checks)¶
A repo passes the Token-ROI gate iff:
METRICS.mdexists and its schema section parses.- Ledger mtime < 7 days whenever AI tools ran in the repo (proves emission, not doc theater).
- Last 50 ledger records carry required fields.
- Unattributed ETC ÷ total ETC ≤ threshold (default 20%).
The gate fails ONLY on missing telemetry or an unattributed spike — NEVER on absolute spend. Blocking merges on a spend number is the folly reinstalled as CI ("Kerr folly squared"). Threshold breach = WARN badge on the board panel.
8. Per-repo METRICS.md template (minimum contract)¶
# METRICS.md — <repo>
standard: token-roi v1.0 # link to this doc
headline: CPDT # formula version
ledger: <path to ledger.jsonl>
attribution: <ticket-id format, e.g. WW-nn> + cwd/branch/window fallback
savings: two-ledger MEASURED/MODELED (never summed)
thresholds: { unattributedPct: 20, cpdtTrend: informational }
seats: [claude, codex, gemini, grok, ollama]
## Current 30d
<generated table: CPDT, $/surviving-commit, rework ratio, unattributed %>
9. Board web-UI panel (required)¶
Per ticket row: attributed $ETC, CPDT contribution, rework ratio, confidence mix. Header gauges: %-5h + tier, %-week, cache hit %, unattributed %, savings (MEASURED $ / MODELED ~$, separate). WARN badges from §7.
10. Per-turn live report (Stop hook)¶
One line after every Claude Code turn, zero fields suppressed:
⛽ $0.19 ET 12.4k (in 8.1k cr 3.2k out 1.1k) | 5h 34% YEL wk 61% | cache 78% | saved $0.04 meas + ~$0.11 est | WW-42
No history, no per-turn ROI (per-prompt impact is banned, §3.5). Tier letter included (governor already computes it).
11. Kerr check — gaming vectors and paired guardrails¶
| Metric | Gaming vector | Paired guardrail |
|---|---|---|
| CPDT ↓ | split tickets tiny | median ticket $ floor + reopen rate |
| CPDT ↓ | starve quality (frugality-max) | rework ratio; review-cycle ETC folds back into CPDT automatically |
| Commit survival | never refactor (protect old lines) | refactor-tag exclusion |
| Rework ratio | never declare "done" | cycle time |
| Cache hit % | pad bloated stable prefixes | $/turn + output/input ratio |
| Savings $ | inflate counterfactuals | method+confidence tags; MODELED never sums into dollars |
| %-limits ↓ | avoid AI when it'd help | throughput of held closes |
| Unattributed bucket | dump bad work there | reported %; rises count against the system owner |
Any NEW metric added anywhere in this shop must ship with its own row in this table before it may appear on a dashboard.
12. Lifecycle-phase spend (v1.1 amendment)¶
Status: council-ratified 2026-07-10 (second sitting, consensus A-D-B-C-E unanimous; report:
llm-council/reports/2026-07-10_132155_design-sitting-per-ticket-lifecycle-phase-spend-.md).
Framing rule: phase spend is descriptive attribution, never value scoring.
12.1 Phase taxonomy — mechanical only, half-open windows [T1, T2)¶
Detected from BoardTransition timestamps joined to ledger timestamps (ticket → branch-map). No self-reporting, no manual classification, ever.
| Phase | Window |
|---|---|
| MAKE | first ledger row tagged to ticket → first →in-progress |
| WORK | union of in-progress intervals before first done |
| REVIEW | union of in-review intervals, per round (→in-review → next done | in-progress) |
| REWORK | everything after first done; each reopened→done = numbered cycle; accumulates forever |
TOTAL@done = MAKE + WORK + ΣREVIEW, frozen at first done, never restated. REWORK sits
beside it, additive, permanent.
12.2 Detection degradation + UNSPLIT¶
- Late board flips (transition timestamp lands after spend already tagged): time-based grace
window (configurable), figures degrade to
estimated. - At most ONE synthesized boundary per ticket: first ledger row on ticket/branch may serve as a
phase start, marked inferred (
estimated). - No usable transitions → UNSPLIT: show TOTAL only, phase cells render
—. Never fabricate a split. (Silently dumping everything into WORK, or inventing "nearest defensible buckets", were both explicitly rejected.) - Confidence reuses the v1.0 enum
exact | measured | estimated | unknown: both boundaries real =exact; synthesized boundary =estimated+~marker. UNSPLIT is a display state, not a confidence value. Rollups take the worst child label. A phase cell degrades tounknownwhen >20% of its window's dollars are unattributed (§4 rule).
12.3 Attribution¶
Join precedence per §4: ticket tag → branch-map → visible unattributed bucket (never rendered on ticket chips). Pre-in-progress = MAKE; in-review windows = REVIEW; post-done = REWORK cycle k; a reopen starts a new cycle and the prior TOTAL stays frozen.
Unmetered seats — floor grammar, mandatory: REVIEW never displays without ≥:
R ≥$0.31 (Claude only; N external seats unmetered). Pair with reviewRounds — rounds are
exactly countable from transitions even when dollars are not. External seats contribute
reviewRounds + unmeteredSeats metadata, never invented dollars.
12.4 Display¶
- Markdown column (one cell):
$2.60 [M .4 | W 1.9 | R ≥.3~] ⟲+$0.90(1)— total, split with~on estimated figures, rework + cycle count. UNSPLIT:$2.60 [unsplit]. One legend footnote per board. - Kanban chip: total right-aligned + 3-segment micro-bar; ⟲ badge only when rework exists, shown forever on done cards; hover = full breakdown + confidence pips + unmetered note.
- Epic rollup: phase sums, N tickets, reopen count, unattributed %, worst child confidence;
≥propagates if any child has unmetered review.
12.5 WARN signals (annotate, never block; never absolute spend)¶
| WARN when | Why |
|---|---|
| ⟲ / TOTAL@done > 1.0 | rework exceeded the build |
| reviewRounds ≥ 3 AND later reopen | rounds alone ≠ WARN — hard tickets earn rounds |
| REVIEW window existed but 0 metered rows | under-instrumentation, not overspend |
| unattributed share > 20% | measuring system broken (§4) |
| all-estimated epic | data-quality WARN, not spend WARN |
Rejected WARNs: Review > 3×Build (punishes thorough review — reinstalls the skimping
incentive); reviewRounds ≥ 1 required before done (a block — violates WARN-never-block).
12.6 Kerr rows (one per displayed figure)¶
| Displayed | Gaming vector | Paired guardrail |
|---|---|---|
| MAKE $ | skimp research to look lean | No WARN on MAKE in either direction, ever. Displayed beside REWORK so high-MAKE/low-REWORK reads as payoff |
| WORK $ | shift spend across phase labels | boundaries mechanical only; late flips degrade confidence, never hand-reclassified |
| REVIEW ≥$ + rounds | skimp review to look efficient | REVIEW always adjacent to rework-within-14d $ on the same row; skimping moves cost to the uglier permanent column. Rounds counted, not judged |
| TOTAL@done | ticket-splitting | v1.0 median floor + reopen rule; Σ phases must reconcile to attributed ETC |
| REWORK ⟲ | never reopen / hide bugs | honest ⟲ renders neutral — no red, no rank. Phantom-rework detector: ledger rows touching a closed ticket's branch/files with no reopen transition → per-epic "phantom rework" bucket, visibly uglier than honest ⟲. Off-board fixes also spike unattributed %. CPDT's 14d rule already punishes hiding at the headline |
| Epic Σ | shuffle spend between epic tickets | joins immutable after row write; no re-bucket UI |
12.7 Rejected from display entirely¶
- Per-phase efficiency ratios / $-per-hour-in-phase — clock-gameable, meaningless solo.
- Cross-ticket phase leaderboards — no guardrail exists.
- Per-seat review $ until PATH shims land — else reviews route to unmetered models to deflate the board.
- Phase-level savings — MEASURED/MODELED stay system-level (§5).
- Phase splits on UNSPLIT tickets — fabricated precision.
12.8 Seam contract (report --board)¶
Per-ticket JSON → dumb generators (markdown + kanban HTML):
1. Generators render only — zero attribution math.
2. JSON emits structured tuples {phase, usd, isFloor, confidence}, never pre-formatted
strings — the presentation layer applies ≥/~.
3. Schema versioned; carries per-figure confidence, cycle index, unmetered flags, unattributed
bucket, boundary-evidence GUIDs, ledgerWatermark/generatedAt.
4. Missing renders —, never $0. Retro board edits append correction events, never rewrite.
12.9 Non-goals¶
No phase savings. No rankings. No manual classification. Retroactive/backfilled tickets are
UNSPLIT by definition (see backfill verb) — historical splits are not reconstructed.