Skip to content

DR / operations — warm pool

Model

  • 4 home Windows PCs: mainpc, gaming, streaming, greatroom.
  • Each keeps 1 ephemeral Linux runner + 1 ephemeral Windows-host runner online at all times (when the PC is up).
  • GitHub is the scheduler. A queued job is routed to one idle matching runner. If none is idle, it requeues until one frees up. No leader election, no lock, no heartbeat service of our own.
  • Runner names are unique: ww-<pc>-<shortguid>. Labels are shared, so any PC's runner with the right OS labels can serve any matching job.
  • Ephemeral = one job per runner. On job completion the runner unregisters itself; the watchdog reaps the exited container/process and launches a fresh one with a clean workspace.

Failure modes and behavior

Failure What happens Operator action
One PC powered off Its 2 runners are simply absent; other 3 PCs serve. GitHub requeues if it briefly had no idle runner. None. Power it back on; watchdog re-registers at logon.
Docker Desktop down on a PC Linux watchdog logs "Docker not available", skips launching; Windows-host runner unaffected. Start Docker Desktop; watchdog self-heals next poll.
Watchdog process crashes Scheduled task restarts it (RestartCount 999, 1-min interval). None.
Job crashes the runner container Container exits non-zero; watchdog reaps it and launches a fresh ephemeral runner. None.
All 4 PCs down Jobs queue on GitHub until a runner appears (or the job times out per GitHub's 24 h queue limit / your job timeout-minutes). Bring any 1 PC online.
Registration token mint fails (network/App) Watchdog logs the error, retries next poll; no runner registered that pass. Check network + App status; see REVOCATION-RUNBOOK if auth-related.
Two runners on one PC (race) Reconcile pass keeps the newest, removes extras. None (self-corrects).

Capacity notes

  • Concurrency across the pool = up to 4 Linux jobs + 4 Windows jobs in parallel (one per PC per OS) at any instant. More PCs online = more parallelism.
  • Because each runner is single-job ephemeral, throughput is (idle runners) × (jobs/hour each). The keep-one-alive design trades peak parallelism-per-PC for isolation + a clean workspace every job.
  • If you need >1 concurrent job per PC per OS, raise the "keep exactly one" count in the watchdog (Get-OurContainers reconcile logic) — but the council chose exactly-one for simplicity and clean isolation.

Routine ops

  • Status across the whole pool: pwsh ./skill/scripts/Runner-Status.ps1 (queries GitHub — shows every PC's runners, not just local).
  • Bring a PC's runners up now: pwsh ./skill/scripts/Runner-Online.ps1 -Once.
  • Take a PC out of the pool: pwsh ./skill/scripts/Runner-Stop.ps1.
  • Bump the pinned runner version: update RUNNER_VERSION (+ checksum) in docker/Dockerfile.linux and -RunnerVersion in Install-WindowsRunner.ps1, rebuild the image, reinstall the host runner, restart watchdogs.

Health signals to watch

  • Runner-Status.ps1 should always show ≥1 idle ww-<pc>-* per online PC.
  • If a runner shows offline in GitHub but the PC is up, the watchdog will reap+relaunch within one poll interval (PollSeconds, default 15s).
  • Persistent busy with no job running usually means a stuck container — the reap logic clears it; if not, Runner-Stop.ps1 then Runner-Online.ps1.