Testing matrix — llm-council¶

This repo follows the house 4-layer testing standard (mocked unit coverage + mutation + churn/flake detection + automation/E2E). It is the fleet's only C# tool, so all four layers are real here — no substitutions.

Pillar	Tool	Status	Gate
Unit	xUnit + Bogus	Implemented (`tests/LlmCouncil.Tests`, 69 tests)	`test` CI job via `tests/Invoke-CI.ps1`
Coverage	coverlet (cobertura)	100% line / 99.0% branch (floor 70%)	`Invoke-CI.ps1` fails under 70%
Mutation	Stryker.NET (real, local dotnet tool)	score recorded below; break threshold 60%	`mutation` CI job, `dotnet stryker`
Churn / flake	dotnet test 5x loop	Implemented (`churn` CI job)	fails if any run's summary differs
Automation / E2E	xUnit driving real `ProcessCliRunner` against `cmd.exe` stubs	Implemented (`ProcessCliRunnerTests`)	runs in the same suite, hermetic

Unit (xUnit + Bogus)¶

The production seam is ICliRunner (src/CliRunner.cs): the council engine never spawns processes directly, it goes through this interface. Tests inject FakeCliRunner — no real AI CLI (claude/codex/agy/ollama) is ever invoked by the test suite. Bogus (seeded, deterministic) fabricates seats, configs, and questions so the logic is exercised with varied data instead of a few hand-picked literals.

Covered: CLI argument parsing, config discovery (explicit path / env var / directory walk-up), tier→model mapping, the 3-stage engine (anonymization labels incl. the 27-seat AA case, seat failure exclusion, chairman-missing and chairman-crash degradation, review failures), prompt construction, report markdown/JSON rendering, slug edge cases.

dotnet test llm-council.slnx
pwsh -File tests/Invoke-CI.ps1     # same + coverage floor

Coverage (100% line / 99.0% branch, floor 70%)¶

tests/coverlet.runsettings excludes exactly one file: src/Program.cs, the composition root. It is ~60 lines of console/stdin/exit-code wiring with no branching logic beyond what CommandLine.Parse (fully tested) returns; testing it would mean spawning the real binary, which the process-level E2E layer already approximates via ProcessCliRunner. Everything else — engine, runner, config, report, parsing — is measured, and the suite holds 100% line / 99.0% branch coverage, far above the 70% house floor enforced by Invoke-CI.ps1. (The floor stays at 70% so honest future refactors don't have to fight a vanity 100% gate.)

Mutation — Stryker.NET (real)¶

Stryker.NET runs as a repo-local dotnet tool (.config/dotnet-tools.json), configured by stryker-config.json:

mutates src/LlmCouncil.csproj except Program.cs (same rationale as coverage);
thresholds: high 85 / low 70 / break 60 — CI fails below 60%.

Honest score: 88.67% (2026-07-08: 225 killed + 2 timeout / 29 survived). The first run scored 66.02%; targeted assertions on report structure, engine status output, truncation boundaries, and the two stdin-pipe failure branches raised it to 88.67% with zero NoCoverage mutants remaining. The 29 survivors are cosmetic: blank-line AppendLine() spacing statements in the markdown report, fragments of [council] ... operator status lines, and process flags (CreateNoWindow, Kill(entireProcessTree)) with no effect observable from a test harness. Killing them would require brittle full-file snapshot asserts; we accept them knowingly.

dotnet tool restore
dotnet stryker

Churn / flake detection¶

The churn CI job runs the full suite 5 times and fails if any run's pass/fail/skip summary differs from the first. The riskiest tests for nondeterminism are the real-process ones (pipe timing, timeout kills); they were designed with wide margins (multi-MB pipe overflows, 3s child vs 1s timeout) to stay deterministic.

Automation / E2E (hermetic)¶

ProcessCliRunnerTests exercises the real process runner end-to-end against cmd.exe builtins standing in for AI CLIs: stdin piping, {prompt}/ {model} substitution, per-tier extra args, %ENVVAR% command expansion, temp-file answer mode (the codex pattern), stderr capture, nonzero exits, missing executables, timeout kills, and both broken-stdin-pipe branches (child never drains / child exits early). No network, no real AI CLIs — safe for CI. Windows-only by design: the tool, its seat CLIs, and the CI runner are all Windows.

A live-CLI smoke (llm-council "ping" --tier haiku against real seats) is deliberately not in CI: it would need authenticated claude/codex/agy sessions on the runner and would burn real quota. Run it manually after config changes.

Local quickstart¶

pwsh -File tests/Invoke-CI.ps1   # unit + E2E + coverage floor
dotnet stryker                    # mutation score + break gate