Skip to content

CMH-7 — Local worker wedge supervisor

Problem (observed live 2026-07-15, not theoretical)

The claude-mem legacy worker (bun.exe running .../claude-mem/<ver>/scripts/worker-service.cjs --daemon, port 37777) wedges: it holds the port in LISTEN, accepts TCP connections, and never answers HTTP.

Measured on the main PC:

Probe Result
http://127.0.0.1:37777/health timeout at 20.1s (wedged)
http://127.0.0.1:37877/ (control) HTTP 200 in 0.17s (alive)

While wedged, ESTABLISHED connections on 37777 climbed 28 → 81 → 115 in minutes (peer-session retry storm). Worker process was idle (8.9s CPU, 25MB WS) — stuck, not busy. Every Claude Code hook that talks to the worker blocks ~60s, which is what the user experiences as "claude-mem crashing constantly".

Killing the exact PID resolved it: a fresh worker respawned and answered HTTP 200 in 0.1s, stuck connections drained to 0.

Upstream thedotmack/claude-mem issues #3204, #3171, #3231 describe exactly this and are ALL OPEN as of 2026-07-15. Local repo is 0 commits behind main (v13.11.0) and already contains the "non-blocking hooks" fix (ce1f5253) — it still wedges. There is no upstream fix to wait for.

Why the existing CMH-5 watchdog does NOT catch this

FailoverEngine handles remote failover (primary → standby claude-mem servers) and reacts to heartbeats. A wedged worker keeps heartbeating ({"status":"worker-running"} every 30s) while answering nothing. Liveness here must be an HTTP probe, not a heartbeat. This is a separate concern and a separate component — do NOT modify FailoverEngine.

Scope

Add a local-worker supervisor to the existing ClaudeMemWatchdog project (extend, do not fork, do not create a new repo). It probes the local worker and, on sustained unresponsiveness, kills the exact wedged PID so the worker respawns clean.

Design (these decisions are settled — implement as specified)

New files in src/ClaudeMemWatchdog/

  1. IWorkerProbe.cs
  2. Task<bool> IsRespondingAsync(string url, CancellationToken ct) — true iff HTTP 200 within timeout.
  3. Must never throw. Timeout / refused / non-2xx → false.

  4. NOTE: a timeout and a 404 are different signals, but for this probe both are "not healthy". Probe /health; if the worker answers any HTTP status quickly it is alive (not wedged) — treat a fast non-200 as ALIVE, a timeout as WEDGED. This distinction is the whole point; encode it explicitly with a WorkerProbeResult enum: Responding, Wedged (timeout), NotListening (connection refused).

  5. HttpWorkerProbe.cs — IWorkerProbe over HttpClient with a configurable timeout.

  6. IWorkerProcessControl.cs

  7. WorkerProcessInfo? FindListenerOn(int port) — resolve the owning PID of the LISTEN socket.
  8. bool IsGenuineWorker(int pid) — see PID-reuse guard below.
  9. bool KillExact(int pid).

  10. bool IsPortBindable(int port).

  11. WindowsWorkerProcessControl.cs — implementation. Resolve the listener owner (GetExtendedTcpTable P/Invoke or equivalent); read command line via WMI/CIM (Win32_Process.CommandLine).

  12. WorkerSupervisorEngine.cs — the tick logic (pure, testable, all I/O behind the interfaces above).

CRITICAL SAFETY — PID-reuse guard (non-negotiable)

We observed the port reported as LISTEN under a dead PID (23144) after the kill. A dead PID can be recycled by Windows and reassigned to an unrelated process. Killing blind by PID would then kill an innocent process.

Before any kill, IsGenuineWorker(pid) MUST verify all of: - process still exists, AND - process name is bun / bun.exe, AND - Win32_Process.CommandLine contains both worker-service.cjs and claude-mem.

If any check fails → DO NOT KILL. Log ERROR with the PID and what was found, and take no action. Never kill by process name. Never taskkill /F /IM. Kill exactly one PID, only after the guard passes.

Real trap from the same box: python/python3 processes present were local.mcpb.roly-blueprint Claude Extensions, not chroma-mcp. Anything that pattern-matches on "looks related" is wrong.

Tick logic (WorkerSupervisorEngine.TickAsync)

probe := IsRespondingAsync(WorkerUrl)
if Responding        -> consecutiveWedged = 0; return
if NotListening      -> consecutiveWedged = 0; return   // nothing to kill; hooks will spawn it
if Wedged            -> consecutiveWedged++
if consecutiveWedged < WorkerFailureThreshold -> return  // default 3, never kill on one bad probe
if now < lastKillUtc + WorkerKillCooldownSeconds -> log INFO "in cooldown"; return
if killsInLastHour >= WorkerMaxKillsPerHour -> log CRITICAL "kill budget exhausted; not killing"; return

info := FindListenerOn(WorkerPort)
if info is null -> log WARN "wedged but no listener owner found"; return
if !IsGenuineWorker(info.Pid) -> log ERROR "refusing to kill PID {pid}: failed identity guard"; return

KillExact(info.Pid)
record kill (time, count)
consecutiveWedged = 0
// VERIFY BY ARTIFACT - do not assume success
after kill: log whether port is bindable OR a new listener answers; log the actual outcome either way

Rationale for the guards: - WorkerFailureThreshold = 3 — a single probe timeout can be a GC pause. Upstream #2996 shows an over-aggressive threshold of 3 in the other direction (spawn cooldown) blocked prompts ~15 min; we are killing, not spawning, so 3 consecutive misses over 3 ticks is the floor, not a hair trigger. - Cooldown + kills-per-hour cap — never kill-loop. If killing does not fix it, thrashing makes it worse. Exhausting the budget must fail loud (log CRITICAL) rather than silently keep killing. - Do not respawn. Claude Code hooks spawn the worker on demand; we verified respawn works unaided (new PID 53720 took the port and answered in 0.1s). Adding a spawner here would race the hooks.

Config additions to WatchdogConfig (all defaulted, back-compat)

public bool   WorkerSupervisorEnabled   { get; set; } = true;
public string WorkerUrl                 { get; set; } = "http://127.0.0.1:37777/health";
public int    WorkerPort                { get; set; } = 37777;
public int    WorkerProbeTimeoutSeconds { get; set; } = 5;
public int    WorkerFailureThreshold    { get; set; } = 3;
public int    WorkerKillCooldownSeconds { get; set; } = 120;
public int    WorkerMaxKillsPerHour     { get; set; } = 5;

Existing configs on disk lack these keys → must deserialize to the defaults above and behave correctly. Validate in Load the same way existing fields are validated (positive ints, non-empty url).

Wiring in Program.cs

Reuse the existing loop and ILog. After the FailoverEngine tick, if WorkerSupervisorEnabled, tick the WorkerSupervisorEngine in the same try/catch style (an error in one must not kill the other). Keep the existing global mutex. Honour --once. Log a startup line with the worker settings, matching the existing startup-log style.

Tests (tests/ClaudeMemWatchdog.Tests/) — hand-rolled doubles, no mocking framework

Follow the existing TestDoubles.cs style. Add FakeWorkerProbe and FakeWorkerProcessControl (recording kills + scripted IsGenuineWorker). Use TimeProvider/FakeTimeProvider as the existing engine tests do. Required cases:

  1. Responding → never kills.
  2. Fast non-200 → treated as ALIVE → never kills.
  3. NotListening → never kills.
  4. Wedged for threshold-1 ticks → no kill; on the threshold-th consecutive tick → kills once.
  5. Wedged, recovers mid-streak, wedges again → counter reset, no premature kill.
  6. Guard: IsGenuineWorker false → NO kill, ERROR logged. (PID-reuse case — the important one.)
  7. No listener owner found → no kill, WARN logged.
  8. Cooldown: second wedge inside cooldown → no kill.
  9. Kill budget exhausted → no kill, CRITICAL logged.
  10. Kill path verifies outcome and logs it (bindable / new listener).
  11. WorkerSupervisorEnabled = false → engine never probes.
  12. Config back-compat: JSON without any Worker* keys loads with the documented defaults.

Constraints

  • C# only (house language policy). .NET version + nullable/analyzer settings: match the existing csproj.
  • Windows-only P/Invoke is acceptable (this is a Windows box) but keep it isolated in WindowsWorkerProcessControl so the engine stays unit-testable and platform-free.
  • Do not touch FailoverEngine, ClaudeMemSwitchExecutor, or SettingsFile.
  • No new NuGet dependencies beyond what the repo already uses (System.Management is acceptable if needed for CIM command-line lookup; prefer it over spawning wmic).
  • Everything must build and all tests must pass: dotnet build + dotnet test.