Skip to content

WorkWingman self-hosted CI runner (warm pool)

Self-hosted GitHub Actions runner infrastructure for the private repo github.com/andrewjonesdev/WorkWingman — and, in org mode, for every repo in an org (e.g. andrewjonesdev-tools) through one shared warm pool. Built to a council-decided architecture: GitHub App auth (no PAT), a leaderless warm pool across 4 home Windows PCs, and ephemeral runners that serve exactly one job each. Per-repo mode is the default; org mode is config-gated (Org in config.psd1 — see docs/ORG-MIGRATION.md).

Security: no secrets live in this repo or in any image. The App private key stays on the host (DPAPI / KeePass vault); containers only ever receive a short-lived runner registration token. See .gitignore and docs/REVOCATION-RUNBOOK.md.

Architecture at a glance

KeePass .kdbx vault  (master copy of each PC's App private key)
        |  one-time per-PC provisioning
        v
Windows DPAPI blob   (runtime cache, bound to this user+machine)
        |  watchdog reads it (HOST only)
        v
App JWT (RS256, ~9 min)
        |  POST /app/installations/{id}/access_tokens
        v
Installation token (~1 h)
        |  POST /repos/{owner}/{repo}/actions/runners/registration-token   (repo mode)
        |  POST /orgs/{org}/actions/runners/registration-token             (org mode)
        v
Runner registration token (~1 h)   <-- ONLY this crosses into a container
        |
        v
docker run  (Linux)  /  config.cmd + run.cmd  (Windows host)
   --ephemeral, unique name ww-<pc>-<shortguid>, shared labels
        |
   serves ONE job, unregisters, exits
        |
   watchdog reaps + relaunches a fresh one  (keep-one-alive)

Scheduler = GitHub. No leader election, no distributed lock. Each of the 4 PCs keeps N idle ephemeral Linux runners (config MaxRunners, default 1, hard cap 4) and ONE idle Windows-host runner. When a job is queued GitHub routes it to one idle matching runner; if none is free it requeues. A down PC just contributes zero runners; the others serve.

Watchdog hardening (org-migration): all GitHub API calls retry with exponential backoff + jitter; the watchdog loops back off (capped 5 min) on consecutive failures instead of hammering every PollSeconds; and the Linux watchdog periodically force-removes this PC's runners that GitHub has reported offline past a threshold (stale-runner sweep). In org mode the shared home-ci label is added automatically.

Duplicate suppression is handled by workflow concurrency groups (see docs/WORKFLOW-CHANGES.md), NOT by runner coordination.

OS matrix — why two runners

WorkWingman CI has a Windows-only backend job (uses System.Security.Cryptography.ProtectedData / DPAPI — Windows only) plus several ubuntu jobs (churn, e2e, frontend, security, docs). So each PC runs both:

Runner Where Labels Serves
Linux ephemeral Docker Desktop (Linux container) self-hosted, workwingman, linux, docker, x64 (+ home-ci in org mode) ubuntu jobs
Windows ephemeral actions/runner on the Windows HOST self-hosted, workwingman, windows, x64 (+ home-ci in org mode) backend (DPAPI) job

The Windows job runs on the host (not a Windows container) — simpler, no nested virt / licensing, and DPAPI/CryptoAPI behave natively. Both are driven by the same watchdog + same App token-minting flow.

File tree

workwingman-ci-runner/
  README.md                        # this file
  .gitignore                       # excludes keys/tokens/config
  config.example.psd1              # per-PC config template (placeholders only)
  docker/
    Dockerfile.linux               # pinned, non-root, ephemeral Linux runner
    entrypoint.sh                  # register(--ephemeral) -> run -> exit
  watchdog/
    GitHubApp.psm1                 # App key -> JWT -> inst token -> reg token
    Watchdog-Linux.ps1             # keep-one-alive loop (Docker)
    Watchdog-Windows.ps1           # keep-one-alive loop (host runner)
  skill/
    skill.md                       # operator skill (online/status/stop/provision)
    scripts/
      Runner-Online.ps1
      Runner-Status.ps1            # queries GitHub /actions/runners (whole pool)
      Runner-Stop.ps1
      Provision-Key.ps1            # PEM -> DPAPI blob, then shred PEM
      Install-WindowsRunner.ps1    # fetch+extract actions/runner (host)
      Register-WatchdogTasks.ps1   # scheduled tasks (at-logon, auto-restart)
  docs/
    SETUP.md                       # GitHub App + per-PC setup checklist
    WORKFLOW-CHANGES.md            # exact runs-on changes WorkWingman needs
    REVOCATION-RUNBOOK.md          # fast revocation
    DR-OPS.md                      # warm-pool operations / failure modes
    MAKING-OF.md                   # living build journal
    BOARD.md                       # ticket-style board (WWCIR-*)
    ORG-MIGRATION.md               # per-repo -> org-level: implemented; deploy checklist
    technical/
      ARCHITECTURE.md              # Mermaid component + sequence diagrams
      SECURITY.md                  # secrets handling, scan results
    plain/
      README.md                   # plain-language sibling of this file
  tests/
    GitHubApp.Tests.ps1            # Pester: module unit tests (mocked GitHub)
    OrgMigration.Tests.ps1         # Pester: org mode, backoff/jitter, stale sweep, bounds
    Repo.Tests.ps1                 # Pester: parse + config + secret scan
    Invoke-CI.ps1                  # single entry point + coverage gate
  TESTING.md                        # 4-layer testing matrix
  .github/workflows/ci.yml          # test / churn / security / secret-gate / e2e-note

Quick start

  1. User does the GitHub App steps once — see docs/SETUP.md §1.
  2. Per PC (docs/SETUP.md §2): copy config.example.psd1 → config.psd1, fill it in; provision the key into DPAPI; build the Linux image; install the Windows host runner; register the watchdog scheduled tasks.
  3. Update WorkWingman workflows runs-on: labels — docs/WORKFLOW-CHANGES.md (do this in the WorkWingman repo; this repo does NOT touch it).

What is verified vs. blocked on the real App

  • Verified locally: PowerShell module + scripts parse (AST) with no syntax errors; the base64url + JWT-assembly logic runs against a locally-generated test RSA key and produces a well-formed, signature-valid RS256 JWT; Dockerfile builds structurally; entrypoint passes bash -n.
  • Blocked on the real App (user action): the live GitHub round-trips — installation-token exchange, registration-token minting, actual runner registration and job pickup. These need the real App ID, installation ID, and private key, which only the user can create. Marked throughout the docs.

Testing

Pester 5+ unit tests (mocked GitHub API, a throwaway in-memory test key, real local DPAPI/Credential Manager round trips, zero live network calls) plus parse/config/secret-scan breadth tests. Full 4-layer matrix, coverage scope, and honest documentation of gaps: TESTING.md.

Security

No secret ever lives in this repo, an image, or CI — see the summary at the top of this file and the full writeup in docs/technical/SECURITY.md (secrets handling, scan results, threat model). Suspect a leak? Go straight to docs/REVOCATION-RUNBOOK.md.

Plain-language version

docs/plain/README.md explains this repo without the jargon, for anyone in the house who wants the "what and why" without the PowerShell.

Org-level migration (implemented; deploy pending)

The council-decided (2026-07-08) move from per-repo to org-level registration is implemented on branch org-migration: set Org in config.psd1 and the same watchdogs mint org-scoped registration tokens, add the shared home-ci label, and serve every repo in the org from one pool. Per-repo mode stays the default and is unchanged. What remains is GitHub-side setup only (App org permission + org install + per-PC config) — the exact checklist lives in docs/ORG-MIGRATION.md.

Stopgap watchdog launch note (2026-07-08): launch Stopgap-Watchdog.ps1 with pwsh 7, not powershell.exe — hidden Windows-PowerShell child processes on this host lack Import-PowerShellDataFile (language-mode quirk) and die before the loop starts.