Skip to content

Testing matrix - caveman-savings

House testing standard: mocked unit coverage + mutation + churn/flake detection + automation/E2E. This document records how each pillar is realized (or why it is N/A) for a PowerShell-only tool.

Pillar Tool Status here Gate
Unit Pester 5+ Implemented (tests/CavemanSavings.Tests.ps1), 23 tests pester CI job
Coverage Pester code coverage Implemented, threshold 60%, actual 66.7% fails build under threshold
Mutation Stryker N/A for PowerShell (see below) substituted by coverage + churn
Churn / flake Pester 5x loop Implemented (churn CI job) fails if any run differs
Automation / E2E N/A - see below Documented gap not applicable to these scripts

Unit (Pester 5+)

tests/CavemanSavings.Tests.ps1 is fully hermetic. Neither script is ever pointed at a real ~/.claude install: every test builds a throwaway fake-home folder under Pester's TestDrive: and overrides $env:CLAUDE_CONFIG_DIR / $env:LOCALAPPDATA to point at it for the duration of the test, restoring the real values afterward. No network access, no live Claude Code session, and - critically - no python on PATH is required, because the compression-skill lookup naturally comes back empty against an empty fake plugin cache.

Covered behavior:

  • both scripts parse with zero errors via [System.Management.Automation.Language.Parser]::ParseFile;
  • neither script contains a bearer-token / API-key / private-key literal (defense in depth alongside the repo-wide secret grep);
  • Apply-CavemanSavings.ps1 throws when the Claude config directory does not exist;
  • the compression step (Step 1) emits the expected warning and skips cleanly when the caveman-compress skill is not installed;
  • the cavecrew rule (Step 2): creates CLAUDE.md from scratch, appends to an existing file that lacks the block, replaces an existing block in place on re-run (idempotent - does not duplicate), and is skipped entirely by -SkipCavecrew;
  • the MCP output cap (Step 3): creates settings.json with the default/custom cap, preserves unrelated existing settings, writes a .bak backup, is idempotent on re-run (does not duplicate the property), and is skipped entirely when -McpOutputCap 0;
  • Migrate-CavemanBackups.ps1's attribution logic: no-op on a missing or empty shared backup dir; moves a backup to its single owning project; resolves a two-candidate ambiguity via matching YAML frontmatter; leaves a backup in the shared slot (with a warning) when attribution is genuinely ambiguous or no project owns it; and removes the shared directory once every backup has been successfully relocated.

Run locally (must be pwsh, not Windows PowerShell 5.1 - see the note below):

pwsh -File tests/Invoke-CI.ps1

pwsh vs Windows PowerShell 5.1

These tests must be run under pwsh (PowerShell 7+), matching the shell: pwsh steps in ci.yml. Under Windows PowerShell 5.1, stream-redirection semantics for Write-Warning inside a script block piped through *>&1 | Out-String behave differently and several Migrate-CavemanBackups.ps1 assertions cannot see the script's console output - this is a 5.1-vs-7 host difference in output-stream capture, not a bug in the scripts themselves. The scripts run correctly under both hosts; only the test harness's output capture needs pwsh. Invoke-CI.ps1 and the CI workflow both invoke pwsh explicitly.

Coverage threshold (60%, actual 66.7%)

The house default coverage floor is ~70%. These scripts fall a little short of that on a pure hermetic unit run because a meaningful slice of Apply-CavemanSavings.ps1 only executes when a real caveman-compress Python package is present and actually invoked (& $python.Source -m scripts $file), plus the mojibake/revert-on-grow branches that follow a real compression run. Exercising those branches would mean either bundling a fake Python skill package in this repo (out of scope - the real skill lives in a separate plugin) or shelling out to a real python interpreter in CI (not hermetic, and not guaranteed present on every self-hosted runner). The floor is set to 60% with the actual measured 66.7% giving comfortable margin; if a future refactor makes the compression step injectable (e.g. accept a -CompressCommand override), coverage should be pushed back toward 70%+ and this note revisited.

Mutation - Stryker is N/A for PowerShell

Stryker has no PowerShell mutator. Stryker.NET (C#), StrykerJS (JS/TS), and Stryker4s (Scala) are the only supported targets. This repo has no C#/JS/TS code, so there is nothing for Stryker to run against. Mutation coverage is substituted by the Pester line/branch coverage threshold above plus the churn (flake) job below.

Churn / flake detection

The churn CI job runs the full Pester suite 5 times and fails if any run's pass/fail/skip counts differ from the first. Verified locally: 5/5 runs at P=23 F=0 S=0, no flake.

Automation / E2E - not applicable

Both scripts are local file-manipulation tools with no network surface, no server to stand up, and no external API to fake - there is nothing for an E2E harness to exercise beyond what the hermetic unit tests already cover end-to-end (each test runs the real script against a real (throwaway) filesystem, just not the developer's actual ~/.claude). This differs from house-council-client, whose scripts register an MCP server over HTTP and therefore warrant a loopback HttpListener E2E test. If a future version of either script gains a network or subprocess dependency worth faking, add a dedicated E2E test then.

Local quickstart

# unit + coverage
pwsh -File tests/Invoke-CI.ps1

# without the coverage floor (what the churn job uses per-iteration)
pwsh -File tests/Invoke-CI.ps1 -MinCoverage 0