Skip to content

Testing matrix - workwingman-ci-runner

This repo is part of the house tool fleet standardized on: mocked unit coverage + mutation + churn/flake detection + automation/E2E. It is a PowerShell-only tool (GitHub App auth, warm-pool watchdogs, skill scripts), so the pillars below follow the same shape - and the same honesty about what is/isn't achievable per language - established by the pilot, house-council-client.

Pillar Tool Status here Gate
Unit Pester 5+ Implemented (tests/GitHubApp.Tests.ps1, tests/OrgMigration.Tests.ps1, tests/Repo.Tests.ps1) test CI job, Invoke-Pester -CI
Coverage Pester code coverage Implemented, threshold 70%, actual ~88% fails build under threshold
Mutation Stryker N/A for PowerShell (see below) substituted by coverage + churn
Churn / flake Pester 5x loop Implemented (churn CI job) fails if any run differs
Automation / E2E - Documented gap (see below) e2e-note CI job is a visible placeholder, not silently skipped

Unit (Pester 5+)

Three hermetic test files, no live GitHub or Docker required:

  • tests/GitHubApp.Tests.ps1 - the depth suite, over watchdog/GitHubApp.psm1 (the one file in this repo that is entirely pure/mockable application logic):
  • ConvertTo-Base64Url - byte-for-byte correctness of the base64url transform.
  • New-WWAppJwt - built and signature-verified against a throwaway, in-memory 2048-bit RSA key generated fresh per test run (RSA.Create(2048)). This key is never registered with GitHub and never leaves the test process except into a TestDrive temp file. A negative control confirms the JWT does not verify against a different key.
  • ConvertFrom-Pkcs1Der (the PowerShell 5.1 ImportFromPem-fallback DER decoder) - called directly via Pester's InModuleScope (it's an internal, non-exported function) and proven correct by a decode -> sign -> verify round trip against the original key, plus a malformed-input negative case.
  • Get-WWAppPrivateKey for all three KeySource values:
    • PlainPemFile - reads the test PEM from a local temp file.
    • DpapiFile - the test PEM is protected with the real System.Security.Cryptography.ProtectedData.Protect (local Windows DPAPI, no network) into a temp blob, then unprotected back through the function.
    • CredentialManager - the test secret is written via cmdkey (the same write path Provision-Key.ps1 -ToCredentialManager uses) into a uniquely-named, test-only Credential Manager entry, read back through Get-WWCredentialSecret, and deleted in AfterAll. All local OS API calls - no network, no GitHub, no Docker.
  • The GitHub token-exchange functions (Get-WWInstallationToken, Get-WWRegistrationToken, Get-WWRemoveToken, Get-WWRepoRunners, Get-WWRunnerRegistration) - the shared HTTP caller Invoke-WWGitHub is mocked at the module boundary (Mock -ModuleName GitHubApp). Every assertion, including the full Get-WWRunnerRegistration key->JWT-> install-token->registration-token flow, runs against canned mock responses. Zero requests ever reach api.github.com.
  • A defense-in-depth scan: the module source contains no PEM private-key block and no live-looking ghp_/gho_/etc. token literal.
  • tests/OrgMigration.Tests.ps1 - the org-migration suite (branch org-migration), same hermetic rules (mocked Invoke-WWGitHub, throwaway test key, zero live calls) plus a no-real-sleeps rule enforced by injectable seams:
  • Org-scope endpoint routing: every runner-scoped function called with -Org is asserted (via Should -Invoke ... -ParameterFilter) to hit the /orgs/{org}/actions/runners/... URI, and with -Repo to still hit the unchanged per-repo URI. Scope validation (repo XOR org, owner/name and org-login shape checks, validation-before-key-load ordering) is covered with negative cases.
  • Backoff + jitter: Get-WWBackoffDelay takes an injectable [Random], so the tests prove exponential growth, the MaxSeconds cap (including huge-attempt overflow safety), jitter bounds, and seeded determinism without any nondeterminism in CI.
  • Retry wrapper: Invoke-WWGitHubWithRetry takes an injectable -SleepAction scriptblock - the tests substitute a delay-recording no-op, so retry/backoff behavior (success-after-transient-failures, giving up at MaxAttempts, refusing to retry a 404) is proven with zero real sleeping. The retryable-vs-fatal status matrix (Test-WWRetryableFailure) and status extraction (Get-WWHttpStatusCode) are tested via InModuleScope.
  • Stale-runner cleanup: Select-WWStaleRunners is pure (fabricated runner listings, a controlled -Now clock, and the first-seen-offline table passed in), covering the full decision matrix (first sighting vs past threshold, back-online pruning, busy protection, other-PC prefix exclusion, vanished-runner table pruning). Invoke-WWStaleRunnerSweep is then tested end-to-end against the mocked API: only THIS PC's long-offline runner is DELETEd.
  • Bounded count: Get-WWRunnerTarget clamping (default/floor/hard-cap/ non-integer) and Get-WWReconcilePlan launch/remove arithmetic.
  • home-ci label injection (Get-WWEffectiveLabels) and config.example.psd1 hygiene (org keys documented but commented; the example stays in per-repo mode).
  • tests/Repo.Tests.ps1 - the breadth suite:
  • AST parse ([Parser]::ParseFile, zero errors) for every watchdog and skill script, so a syntax error anywhere in the fleet fails CI even though those scripts aren't coverage-instrumented (see "Coverage scope" below).
  • docker/entrypoint.sh basic shell-contract sanity (shebang, set -euo pipefail).
  • config.example.psd1 parses, contains every key the watchdogs read, has only placeholder AppId/InstallationId, and its LinuxLabels/ WindowsLabels match the label sets documented in docs/WORKFLOW-CHANGES.md.
  • A repo-wide secret-literal scan that mirrors the CI hard gate (same regex alternation as the secret-gate CI job) - so a leak is caught by Invoke-Pester locally, not just by CI.

Run locally:

Invoke-Pester -CI

Coverage scope and threshold (70%, actual ~88%)

tests/Invoke-CI.ps1 instruments only watchdog/GitHubApp.psm1, not Watchdog-Windows.ps1, Watchdog-Linux.ps1, or the skill/scripts/*.ps1 launchers. Why, so the next fan-out repo calibrates correctly:

  • GitHubApp.psm1 is the one file built entirely of pure functions (JWT assembly, DER decoding, key loading, HTTP-call composition) that can be exercised hermetically with a test key and mocks. It genuinely earns ~81% branch/line coverage this way - not padding.
  • Watchdog-Windows.ps1 / Watchdog-Linux.ps1 are top-level scripts whose entire body is function defs... ; while ($true) { reconcile; sleep }. There is no -Once-only entry point that skips the infinite loop cheaply enough to dot-source for coverage, and their reconcile functions call docker, Get-CimInstance, and Start-Process directly (not through an injectable seam) - mocking those inline would require refactoring runtime behavior, which this pass explicitly does not do. They ARE parse-tested and secret-scanned in tests/Repo.Tests.ps1.
  • skill/scripts/*.ps1 are thin CLI launchers over the watchdog + module functions already covered; they're parse-tested for the same reason.
  • Within GitHubApp.psm1 itself, two small gaps remain and are intentional, not oversights:
  • Invoke-WWGitHub's own body (header assembly + Invoke-RestMethod call) never executes for real - it's mocked, per house policy that tests never make a live GitHub call. Its correctness is instead proven end-to-end by asserting the mocked calls happened with the right method, URI, and auth header (Should -Invoke ... -ParameterFilter).
  • The manual PKCS#1 DER-decode branch inside Get-WWAppPrivateKey (used only on Windows PowerShell 5.1, which lacks RSA.ImportFromPem) doesn't execute under pwsh 7 - which is what this repo's CI runs - because pwsh 7's RSA has ImportFromPem and takes that branch instead. ConvertFrom-Pkcs1Der itself (the interesting logic in that branch) IS separately and directly unit-tested via InModuleScope, independent of which branch a given host takes.
  • The Get-WWAppPrivateKey -KeySource CredentialManager integration test (multi-line PEM stored via cmdkey) self-skips with Set-ItResult -Skipped if cmdkey can't hold a multi-line secret on the host running the tests - a known Windows cmdkey argument-length/newline quirk, not a code defect. The simple (single-line-secret) Credential Manager round trip is NOT skipped and always runs.

With that scope, GitHubApp.psm1 measures ~88% line coverage (~81% before the org-migration pass; the new org/backoff/stale/bounds functions were built pure-with-injectable-seams precisely so they'd be fully coverable), clearing the 70% floor with real margin (higher than the pilot's 40%, because this module has substantially more pure logic than the pilot's install/bootstrap scripts). The org-migration additions to the watchdog scripts themselves (scope selection, loop backoff, sweep cadence) stay thin by design - all the decision logic they call lives in the instrumented module. Rule for the fan-out repos: any repo with C#, JS, or TS keeps the house 70%+ floor with real Stryker; this PowerShell-only shape's 70% (vs. the pilot's 40%) reflects that this module's logic is unusually mockable, not a relaxed standard.

Mutation - Stryker is N/A for PowerShell

Stryker has no PowerShell mutator (Stryker.NET / StrykerJS / Stryker4s cover C#, JS/TS, and Scala only). We do not fake a mutation score here.

  • This repo (PowerShell): mutation coverage is substituted by the Pester line/branch coverage threshold (70% floor, ~81% actual) plus the churn job.
  • Rule for the fleet: any repo containing C#, JS, or TS must run real Stryker.NET / StrykerJS with a real break threshold. This N/A applies only to PowerShell-only tool repos like this one.

Churn / flake detection

The churn CI job runs the full Pester suite 5 times and fails if any run's pass/fail/skip counts differ from the first - catching nondeterminism (e.g. a TestDrive cleanup race, or a Credential Manager write/read ordering assumption) that a single run would hide.

Automation / E2E - documented gap, not silently skipped

Unlike the pilot (house-council-client, which could stand up a real HttpListener loopback mock because its scripts talk to a local Ollama-shaped HTTP API), this repo's actual external dependency is GitHub's App installation-token and registration-token endpoints, which require a real App ID, installation ID, and private key that only the user can create (see README "What is verified vs. blocked on the real App"). A fully hermetic E2E would either:

  1. Fake the entire GitHub REST surface behind a loopback HttpListener (more integration-test-shaped mocking than E2E), or
  2. Require the real App credentials in CI, which this repo's security model explicitly forbids (no secrets in CI - see docs/technical/SECURITY.md).

Rather than silently -Skip this pillar, the e2e-note CI job runs as a visible, always-green placeholder that states the blocker in its log, so the gap shows up in CI history instead of disappearing. The org-migration CODE has landed (branch org-migration), but the real App/installation is still a user-side deploy step (see the checklist in docs/ORG-MIGRATION.md); once that exists, this job should be replaced with a real registration round trip (WWCIR-4).

The unit suite's mocked Get-WWRunnerRegistration test (see above) is the closest hermetic approximation available today - it proves the composition of the four-hop auth flow is correct, just not against the real API.

Local quickstart

# unit + coverage
pwsh -File tests/Invoke-CI.ps1

# without the coverage floor (what the churn loop does per-iteration)
pwsh -File tests/Invoke-CI.ps1 -MinCoverage 0

# just the breadth suite
Invoke-Pester -Path tests/Repo.Tests.ps1 -CI