Skip to content

Testing matrix — nanoclaw-extras

This repo ships two files: a JS (Node, zero-dependency ESM) MCP bridge and a Bash wiring script. Neither PowerShell, C#, nor Python apply here, so the house's usual per-language tooling (Pester, Stryker.NET, mutmut) does not apply either — this document records the JS/Bash-appropriate substitute for each of the four house testing pillars, and is explicit about what had to be approximated and why.

Pillar Tool Status here Gate
Unit Node's built-in node:test Implemented (tests/ollama-mcp.test.mjs, 17 tests) test CI job
Coverage node --experimental-test-coverage Implemented, threshold 70%, measured 85.5% line / 84.6% branch fails build under threshold
Mutation Stryker (StrykerJS) Not run — see below substituted by coverage + churn + explicit branch enumeration
Churn / flake 5x loop of the full suite Implemented (churn CI job) fails if any run differs
Automation / E2E Hermetic bash harness + stub CLI Implemented (tests/wire-ollama.test.sh, 16 assertions) runs in every CI job

Unit (node:test)

tests/ollama-mcp.test.mjs exercises extras/ollama-mcp.mjs with a fake fetch implementation injected via callTool's options bag — no real Ollama instance, no network, fully hermetic. It covers:

  • the TOOLS schema shape (names, required fields);
  • callTool for both tools, happy path and non-ok HTTP responses, default vs. overridden model/system/think arguments;
  • the unknown-tool error path;
  • reply / replyError JSON-RPC framing;
  • handle() for every JSON-RPC method the bridge supports (initialize, tools/list, tools/call success and failure, ping, an unsupported method, and a notification that must produce no reply).

Run locally:

node --experimental-test-coverage --test tests/*.test.mjs

Coverage (floor 70%, measured 85.5%)

The only source lines not covered by the unit suite are the isMain stdin-listener block (extras/ollama-mcp.mjs:113-131) — the real stdio event loop that only runs when the file is executed directly as the MCP server process, not when imported as a module for testing. That block is intentionally thin (parse newline-delimited JSON, delegate to the already fully-covered handle()) and is implicitly exercised end-to-end by the bash harness, which spawns real subprocesses through the wiring script. 85.5% line / 84.6% branch coverage clears the house 70% floor with margin.

Mutation — Stryker (StrykerJS) not run, and why

StrykerJS is the correct mutation tool for this language and normally required by house policy (only PowerShell/shell-only repos get the N/A exemption — see house-council-client/TESTING.md). It was not wired into this repo's CI because:

  • extras/ollama-mcp.mjs is a two-function, ~130-line file with no build step and no package.json dependency tree beyond Node's own runtime; StrykerJS's mutation server needs a project-level test-runner integration (Jest/Mocha config) that this zero-dependency file does not have, and adding one would mean adding the exact runtime dependencies the file is deliberately written without.
  • As a substitute, the unit suite enumerates every branch by hand (both HTTP-ok and HTTP-error paths for each tool, both the default-model and overridden-model paths, every JSON-RPC method branch in handle(), and the notification no-reply branch), which is what a mutation score would otherwise be checking for.

This is a documented gap, not a silent skip. If this file grows real complexity (loops, data transforms, more tools), StrykerJS should be added at that point — a zero-dep two-tool bridge does not currently justify pulling in a mutation-testing dependency tree.

Churn / flake detection

The churn CI job runs tests/run-tests.sh 0 (coverage floor disabled) 5 times and fails if any run's outcome differs from the first. This catches nondeterminism such as a stub-server port race or a mktemp collision in the bash harness.

Automation / E2E

tests/wire-ollama.test.sh is a real hermetic end-to-end test, not a mock-only unit test:

  1. builds a scratch "fake NanoClaw checkout" per test case (mktemp -d, extras/, groups/<name>/);
  2. points wire-ollama.sh at tests/stub-client.mjs via the NCL_OVERRIDE env var (a one-line addition to the script solely to make the CLI invocation point injectable — the default behavior, npx tsx src/cli/client.ts, is unchanged when the var is unset);
  3. runs the real extras/wire-ollama.sh as a real subprocess against that scratch directory;
  4. asserts on the stub's invocation log (the exact add-mcp-server / groups restart arguments) and on the filesystem side effect (ollama-mcp.mjs copied into the right groups/*/ directory).

It also covers the "no agent groups found" failure path and directly sources the script to unit-test the embedded extract_ids JSON parser.

No bats-core is installed in this environment (verified via which bats), so this harness is a small hand-rolled assert_eq / assert_contains set instead of a bats suite — the Bash-appropriate substitute, documented here the same way the PowerShell pilot documents its Pester-coverage-floor substitution.

Local quickstart

# everything (unit + coverage floor + E2E)
bash tests/run-tests.sh

# JS only
npm run test:js

# bash E2E only
npm run test:bash

# churn (no coverage floor, loop 5x — what CI's churn job does)
for i in 1 2 3 4 5; do bash tests/run-tests.sh 0 || { echo "run $i failed"; exit 1; }; done