Skip to content

Testing matrix - house-council-client

This repo is the pilot template for ~9 more house tool repos. The house testing standard is: mocked unit coverage + mutation + churn/flake detection + automation/E2E. This document records how each pillar is realized (or why it is N/A) for a PowerShell tool, so the fan-out repos inherit the same matrix and the same honesty about what is/ isn't achievable per language.

Pillar Tool Status here Gate
Unit Pester 5 Implemented (tests/Council.Tests.ps1) pester CI job, Invoke-Pester -CI
Coverage Pester code coverage Implemented, threshold 40% fails build under threshold
Mutation Stryker N/A for PowerShell (see below) substituted by coverage + churn
Churn / flake Pester 5x loop Implemented (churn CI job) fails if any run differs
Automation / E2E Pester + System.Net.HttpListener mock Implemented (tests/ConnectE2E.Tests.ps1) runs in CI (see gating note)

Unit (Pester 5)

tests/Council.Tests.ps1 is hermetic - no live council server needed. It asserts:

  • each script parses with zero errors via [System.Management.Automation.Language.Parser]::ParseFile;
  • no council_ bearer token literal exists in any script (defense in depth against a future re-leak of the scrubbed secret);
  • the IPv4 resolver regex extracts 10.0.0.7 from a sample ping.exe output line.

Run locally:

Invoke-Pester -CI

Coverage threshold (40%, not 70%)

The house default coverage floor is ~70%. These connect scripts cannot honestly hit 70%, and the threshold is deliberately set to 40% here. Why, so the fan-out repos calibrate correctly:

  • Each script embeds a large SKILL.md here-string (~40 lines) that Pester counts as executable commands. That block only runs during real skill install (writing into the user's ~/.claude profile), which the hermetic E2E deliberately skips via -SkipSkillInstall. It inflates the denominator with lines that cannot be covered without mutating the developer's machine.
  • The mDNS resolver / curl.exe happy-eyeballs fallback branches only fire against the real house network; they are not reachable from a loopback mock.

With the hermetic unit + E2E suite the two connect scripts measure ~45% line coverage, which clears the 40% gate with margin. This is a language/shape limit, not a slack standard - for repos in this fleet that contain real application logic (C#/JS/TS), keep the house 70% floor. Only install/bootstrap shell scripts get the reduced floor, and only with this documented rationale.

Mutation - Stryker is N/A for PowerShell

Stryker has no PowerShell mutator. Stryker.NET (C#), StrykerJS (JS/TS), and Stryker4s (Scala) are the only supported targets. We do not fake a mutation score here.

  • This repo (PowerShell): mutation coverage is substituted by the Pester line/branch coverage threshold plus the churn (flake) job.
  • Rule for the other fleet repos: any repo that contains C#, JS, or TS must run real Stryker.NET / StrykerJS with a real mutation-score threshold. Do not carry this "N/A" over to a repo that has mutatable application code - it applies only to shell-only tools.

Churn / flake detection

The churn CI job runs the full Pester suite 5 times and fails if any run's pass/fail/skip counts differ from the first. This catches nondeterminism (e.g. a port-bind race in the mock server, or an ordering assumption) that a single run would hide.

Automation / E2E

tests/ConnectE2E.Tests.ps1 is a real hermetic end-to-end test:

  1. starts a System.Net.HttpListener on a random loopback port that answers /health with a fake {"models":[...]} JSON and 200;
  2. shims the claude CLI with a claude.cmd on PATH that captures its arguments;
  3. runs the connect script with -HostOverride 127.0.0.1:<port> -SkipSkillInstall;
  4. asserts the script invoked claude mcp add with the correct /mcp URL and the Authorization: Bearer <token> header.

This required two small, test-only script parameters (-HostOverride, -SkipSkillInstall) that have no effect on normal runs.

CI gating decision

The E2E test is not skipped in CI - it is hermetic (binds loopback, no external network, no real claude) and passes on a self-hosted Windows runner. It is gated behind the RUN_E2E environment variable, which the pester CI job sets to 1. If a future runner blocks loopback HttpListener binding, unset RUN_E2E and the Describe self-skips with this documented reason (needs a bindable loopback endpoint) rather than failing the build.

Local quickstart

# unit + E2E
$env:RUN_E2E = "1"
Invoke-Pester -CI

# with coverage
$cfg = New-PesterConfiguration
$cfg.Run.Path = 'tests'
$cfg.CodeCoverage.Enabled = $true
$cfg.CodeCoverage.Path = 'scripts/connect-gaming-pc-council.ps1','scripts/connect-streaming-pc-council.ps1'
Invoke-Pester -Configuration $cfg