Skip to content

Cancel lane (in-flight job cancel)

Status: implemented (FLT-16 follow-up). Consumer: FMC cockpit POST /api/agent-runs/{jobId}/cancel (fleet-mission-control), which previously could only delete a QUEUED job's inbox file and returned 409 AlreadyRunning for in-flight jobs.

Why a separate lane

QueueService.ListenOnceAsync drains the inbox serially (await _executor.ExecuteAsync inline), so a "kill" job enqueued behind a running job cannot execute until that job finishes. Cancellation therefore travels outside the inbox entirely: a signed marker file that the listener polls during a run.

Marker

Path: <dispatchRoot>/<TARGET>/cancel/<jobId>.json

{
  "jobId":     "<the dispatched job's id>",
  "target":    "<TARGET machine name>",
  "createdAt": "<ISO-8601 UTC>",
  "origin":    "<sender machine name>",
  "sig":       "<hex HMAC-SHA256>"
}

Signature: HMAC-SHA256 with the fleet key (~/.claude/fleet-dispatch.key) over the canonical string cancel|{jobId}|{target}|{createdAt}|{origin}, lowercase hex — same key and posture as JobSigner. OneDrive write access alone must never be enough to kill a run.

Listener acceptance rules (CancelService.IsCancelRequested): - file name must be <jobId>.json AND the body's jobId/target must match — a renamed marker cannot retarget a cancel; - signature must verify; unsigned/garbled markers are ignored and swept after 24h (CancelService.MaxMarkerAge, file write time, not the signer-controlled createdAt); - job ids match ^[A-Za-z0-9-]{1,64}$ and targets are hostname-shaped (^[A-Za-z0-9][A-Za-z0-9_-]{0,62}$) before touching any path; the fleet allowlist check (TargetResolver.Resolve) sits at the CLI/sender boundary, mirroring enqueue; - markers over 16 KiB (CancelService.MaxMarkerBytes) are refused before being read.

Replay is harmless by construction: job ids are single-use and AlreadyProcessed blocks re-runs, so a re-dropped marker can only "cancel" a job that no longer exists.

Listener behavior

Both execution paths (queue drain and serve HTTP) wrap the executor in CancelService.ExecuteWithCancelAsync:

  • a watcher polls the marker every 2s (PollInterval) and, on a valid marker, cancels the per-job linked token — the existing timeout plumbing then tree-kills the process (Windows job object in RealProcessRunner, Kill(entireProcessTree) in RealJobExecutor);
  • the result is rewritten to ok=false, exitCode=-1, error="cancelled" and the job file still moves to archive/done (replay guard holds); the marker moves to archive/consumed/<jobId>.cancel.json;
  • completion race: if the process finishes successfully before the kill lands, the honest success result is kept and the marker is consumed anyway;
  • a marker seen while the job is still queued cancels it before start (error="cancelled-before-start", nothing spawned) — on BOTH paths (queue drain and HTTP dispatch), and the check runs before the governor gate, so a cancelled job never sits in the inbox as "deferred" through a RED tier;
  • the queue rejects jobs whose target field doesn't match the listening machine (wrong-target) — a misdelivered job's cancel markers would live under the other target's directory, splitting the cancel lane.

ListenSummary gained a Cancelled counter (kills + before-start skips; a raced completion counts as Done).

Sending a cancel

CLI: fleet-dispatch cancel --target PC --job-id <id>

Cockpit: mint the same marker (it already holds the fleet key server-side) — on 409 AlreadyRunning, write the marker instead and report "cancel requested"; the run dies within one poll interval plus process-kill time. Queued-job cancel can keep the existing inbox-file delete, or write a marker (belt-and-braces: the marker also covers the delete-vs-sync race).