Skip to content

Testing matrix - claude-mem-house

This repo is part of the house tool-repo fan-out (pilot: house-council-client). The house testing standard is: mocked unit coverage + mutation + churn/flake detection + automation/E2E. This document records how each pillar is realized (or why it is N/A) for this PowerShell tool, and is explicit about a real gap: the coverage floor here is 0%, substituted, for reasons specific to these scripts' shape - read the section below before assuming that means "untested."

Pillar Tool Status here Gate
Unit (static) Pester 5/6 Implemented (tests/Scripts.Tests.ps1) pester CI job
Unit (behavioral) Pester 5/6, child-process Implemented (tests/Guards.Tests.ps1) pester CI job
Coverage Pester code coverage Floor 0%, substituted (see below) documented, not silently skipped
Mutation Stryker N/A for PowerShell (see below) substituted by behavioral tests + churn
Churn / flake Pester 5x loop Implemented (churn CI job) fails if any run differs
Automation / E2E real scripts against a mock server Deferred, documented (see below) not implemented this pass

Unit - static (tests/Scripts.Tests.ps1)

Hermetic, no live server, no Docker. Asserts:

  • each of the 4 scripts parses with zero errors via [System.Management.Automation.Language.Parser]::ParseFile;
  • no cmem_<20+ chars> API-key literal exists in any script, including archive/;
  • the retired Postgres password (scrubbed during this repo's build) never regresses back into any script - a literal tripwire, concatenated so the test file itself can't trip the repo-wide secret grep;
  • claude-mem-client-setup.ps1 and claude-mem-standby-setup.ps1 read their secrets from $env:CLAUDE_MEM_HOUSE_API_KEY / $env:CLAUDE_MEM_HOUSE_PG_PASSWORD (parameter defaults), not from inline literals;
  • the sync script that claude-mem-standby-setup.ps1 writes via a here-string to ~\.claude-mem\claude-mem-standby-sync.ps1 is itself free of the password literal and reads it from the environment / the standby's own .env at run time.

Unit - behavioral (tests/Guards.Tests.ps1)

These tests run the real .ps1 files - not a mock, not a rewritten copy - and assert their actual abort behavior:

  • claude-mem-client-setup.ps1 exits non-zero and prints the CLAUDE_MEM_HOUSE_API_KEY guidance when the API key is unset, before it ever reaches the health-check step;
  • claude-mem-standby-setup.ps1 exits non-zero for a missing Postgres password, before the primary-machine check;
  • claude-mem-standby-setup.ps1 refuses to run when -PrimaryHost resolves to this machine, before any prerequisite-install step (git/node/bun/docker);
  • claude-mem-switch-server.ps1 exits non-zero when no API key is resolvable from -SettingsPath or the environment, before the target-server health check.

Why child processes, not in-process & $script: the pilot's E2E test (ConnectE2E.Tests.ps1) calls its connect scripts in-process because their happy path never calls exit. These guard tests exist specifically to prove an exit 1 abort path works - and PowerShell's exit statement terminates the entire process, even when the script was invoked via the call operator from inside another script. Running the guarded scripts in-process would kill the whole Pester session on the first assertion. Each test therefore spawns the real script as a child powershell.exe -File process (env vars cleared/overridden per case) and asserts on that child's exit code and captured output. This is real behavioral verification of the actual shipped scripts, not a rewrite.

Coverage floor: 0%, substituted

The house default coverage floor is ~70%, reduced to 40% for the pilot's connect scripts, and reduced further here to an honest 0% SPECIFICALLY because of the child-process constraint above:

  • Pester's -CodeCoverage engine instruments the current process via breakpoints. It cannot see anything that executes inside a child powershell.exe process, which is exactly how tests/Guards.Tests.ps1 must invoke these scripts (previous section).
  • The static tests in Scripts.Tests.ps1 never execute the scripts at all (ParseFile parses without running; the secret-literal checks are plain text regex over the file content) - so they contribute 0 executed lines too.
  • Net result: pwsh -File tests/Invoke-CI.ps1 reports 0% line coverage for scripts/*.ps1, even though 19/19 tests pass and every guard clause is exercised against the real, shipped script.

This is a stricter version of the pilot's documented "language/shape limit, not a slack standard" rationale - these are full standalone installer/failover CLI entry points (elevated installs, Docker, scheduled tasks, DNS resolution, Postgres dumps) with almost no code reachable without a real house network, a real Docker daemon, and admin rights. Rather than inflate the number with a coverage-only rewrite that doesn't test the real artifact, the substitute is: parse-clean (structural correctness) + secret-literal regression tests (security) + real child-process behavioral guard tests (functional correctness of every abort path) + churn (flake detection across all of the above). Rule for other fleet repos: a 0% floor is only acceptable with this documented constraint (exit-driven guard clauses needing child-process testing); do not carry it over to a repo with mutable/coverable application logic.

Mutation - Stryker is N/A for PowerShell

Stryker has no PowerShell mutator. Stryker.NET (C#), StrykerJS (JS/TS), and Stryker4s (Scala) are the only supported targets. We do not fake a mutation score here.

  • This repo (PowerShell): mutation coverage is substituted by the child-process behavioral guard tests (which fail if a guard clause's logic or message changes) plus the churn (flake) job.
  • Rule for the other fleet repos: any repo that contains C#, JS, or TS must run real Stryker.NET / StrykerJS with a real mutation-score threshold. Do not carry this "N/A" over to a repo that has mutatable application code - it applies only to shell-only tools.

Churn / flake detection

The churn CI job runs the full Pester suite 5 times and fails if any run's pass/fail/skip counts differ from the first. Verified locally: 5/5 identical (P=19 F=0 S=0) - the child-process guard tests are deterministic (no shared mutable state, unique temp paths per run, network never touched).

Automation / E2E - deferred

Unlike the pilot's connect scripts, a full E2E here would need a mock claude-mem HTTP server (/healthz, /v1/jobs, /v1/mcp, /v1/search) and either a real or container-mocked Docker/Postgres for the standby path - the setup scripts' happy paths are inseparable from Docker compose and winget/bun.sh installs, which is a materially larger hermetic-mock surface than the pilot's single /health endpoint. This pass ships the guard-clause behavioral tests (above) as the achievable equivalent and defers a full happy-path E2E (HttpListener mock of /healthz + /v1/jobs + /v1/mcp + /v1/search, run against claude-mem-client-setup.ps1 only, since it has no Docker dependency) as follow-up work - tracked in docs/BOARD.md.

Local quickstart

# full suite
Invoke-Pester -CI

# with coverage (reports 0% - see "Coverage floor" above)
pwsh -File tests/Invoke-CI.ps1

# churn (5x, no coverage gate)
for ($i=1; $i -le 5; $i++) { Invoke-Pester -CI }