CI: self-hosted Windows runner provisioning¶

Plain-language version: Why the Robots Kept Failing

Diagnosed 2026-07-15. Master CI had been red since ~07-12. None of it was bad code — gitleaks and semgrep both passed locally against master with the exact CI invocations (exit 0, zero findings). Every failure was infrastructure.

Status 2026-07-16: causes 2 and 3 are FIXED and verified. Cause 1 (billing) is not. The prescriptions in this doc were carried out; the per-cause detail below records what each symptom was and how it was resolved, because the pattern keeps recurring and is worth keeping written down. Every fix was an account-scoping problem, not code: anything installed or configured per-user on this box is invisible to the runner.

Proof the provisioning work is done — the Backend job, which for days died at step 3, now reaches its own test suite:

3  Install .NET 10 SDK          success   <- DOTNET_INSTALL_DIR (in-repo, PR #16)
4  Build solution               success   <- ExecutionPolicy RemoteSigned (host)
5  Vulnerable package gate      success
6  Install Playwright browser   success   <- machine-wide PowerShell 7 (host)
7  Run backend tests            1 failed, 3635 passed

The one remaining failure is LocalRagLiveSmokeTests — a live-Ollama smoke test whose service the runner account cannot reach. Same account-scoping shape as everything below; it is a live-service dependency inside the standard suite, and it is a separate decision (see "Still open").

Three independent causes¶

1. Billing — kills the 3 `ubuntu-latest` jobs¶

Semgrep (SAST), gitleaks (secrets), and ZAP (DAST) never get a runner: runner_name: "", 0 steps, ~5s, instant failure. GitHub's annotation:

The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings

WorkWingman is a private repo, so it consumes GitHub-hosted minutes. Requires an account/billing action — nothing in this repo can fix it.

2. No self-hosted Linux runner — 8 jobs queue forever — FIXED 2026-07-16¶

Eight jobs request [self-hosted, workwingman, linux, docker, x64] (Frontend, deps, churn, diagrams, E2E…). At diagnosis the repo had exactly one registered runner: ww-DESKTOP-D249KD4-win (Windows). Those jobs never started.

A Linux runner now exists and serves the repo. gh api .../actions/runners → total_count: 2: ww-andygreatroompc-linux (online, [self-hosted,X64,workwingman,Linux,docker]) alongside the Windows one. Verified by real runs — docs, churn, and frontend jobs all executed on it. Those jobs no longer queue forever; they run and fail on merit, which is the point. (Cause 3's DOTNET_INSTALL_DIR fix had to be extended to the Linux .NET jobs once they started running — they hit the identical permission wall in its Linux wording: mkdir: cannot create directory '/usr/share/dotnet': Permission denied.)

Note: the home-ci runner group lives under the andrewjonesdev-tools org, but this repo lives under the andrewjonesdev user — org runners do not serve it.

Side effect: because those jobs sit queued, the whole workflow run never completes, so gh run view --log reports "still in progress" forever. Use the per-job log API instead: gh api repos/{owner}/{repo}/actions/jobs/{job_id}/logs.

3. The Windows runner is under-provisioned for the Backend job¶

The backend job's own comment still says "This job runs on windows-latest" — it was moved to the self-hosted runner, which is not provisioned to do its work.

The runner service runs as NT AUTHORITY\NETWORK SERVICE. That single fact explains every symptom:

Symptom	Cause
`setup-dotnet` fails: "current user doesn't have write access to the installation root 'C:\Program Files\dotnet'"	NETWORK SERVICE is not an admin. Fixed in-repo via `DOTNET_INSTALL_DIR: ${{ runner.tool_cache }}/dotnet` — verified `setup-dotnet@v4` honours it (`DotnetInstallDir.dirPath = process.env['DOTNET_INSTALL_DIR']`, then `addToPath`/`exportVariable` derive from it), so the pinned `10.0.x` guarantee holds. Step 3 now passes.
`Build solution` fails: `PSSecurityException` / `UnauthorizedAccess` / fwlink `LinkID=135170`	`Get-ExecutionPolicy -List` was Undefined at every scope → effective Restricted, so the runner could not dot-source the `.ps1` it generates per `run:` step. FIXED 2026-07-16 (host): `Set-ExecutionPolicy RemoteSigned -Scope LocalMachine`. `LocalMachine` is load-bearing — `CurrentUser` would apply to the interactive login and do nothing for NETWORK SERVICE.
`shell: bash` does not help	`bash` on PATH is `C:\Windows\System32\bash.exe` — the WSL stub, and WSL is not installed (fails with `wsl.exe --install <Distro>`). Git Bash exists at `C:\Program Files\Git\bin\bash.exe` but is not on PATH. Still true; irrelevant now that PowerShell works.
`pwsh` (Playwright step) will fail for the same account	Correct, and it did — `pwsh : The term 'pwsh' is not recognized`. `pwsh` resolved only via a per-user WindowsApps shim (`C:\Users\AndrewJonesDev\AppData\Local\Microsoft\WindowsApps\pwsh.exe`) under the interactive profile; NETWORK SERVICE has no such profile. FIXED 2026-07-16 (host): PowerShell 7.6.3 installed machine-wide to `C:\Program Files\PowerShell\7\` + SYSTEM PATH; verified by running `pwsh` as NETWORK SERVICE via a scheduled task (`PROBE_OK 7.6.3`). Swapping the workflow to `powershell` 5.1 is not an alternative: `playwright.ps1` does `[Reflection.Assembly]::Load(Microsoft.Playwright.dll)` from `bin/Release/net10.0/`, and 5.1 runs on .NET Framework 4.8, which cannot load a .NET 10 assembly. The `#!/usr/bin/env pwsh` shebang is a real requirement.

What fixed it — option B was taken, 2026-07-16¶

A. Restore billing and move these jobs back to GitHub-hosted runners. Would fix causes 1 and 3 together. Not taken — but note cause 1 still stands on its own: the three ubuntu-latest scanners remain dead until billing is restored, and no repo change can fix that.

B. Provision the self-hosted runner properly. ✅ Done. What was actually carried out, against the original checklist:

~~Run the runner service as a non-NETWORK SERVICE account~~ — not needed. It still runs as NETWORK SERVICE. DOTNET_INSTALL_DIR (in-repo, PR #16) removed the need to write privileged paths at all, which is the cheaper half of this item and did not require touching the service identity.
✅ Set-ExecutionPolicy RemoteSigned -Scope **LocalMachine**.
✅ PowerShell 7.6.3 installed machine-wide via MSI.
✅ A Linux/docker runner is registered (ww-andygreatroompc-linux).

Workflow-level workarounds (shell: cmd, -ExecutionPolicy Bypass) were considered and rejected: they paper over a misprovisioned host, and could not have fixed the missing pwsh for the service account anyway — the job would still have gone red, one step later. That call held up: the pwsh failure landed exactly as predicted here, and only the machine-wide install cleared it.

Two traps worth remembering¶

winget --scope machine reported success and installed nothing. It printed Successfully installed while C:\Program Files\PowerShell\7\pwsh.exe did not exist — almost certainly a user-scope Store package again, i.e. the very bug being fixed, reported as a win. Only an artifact check caught it; the MSI fallback did the real work. Check the file, never the exit code.

Verifying as yourself proves nothing here. The entire failure mode is "works for the interactive user, missing for the service account". Any check must run as NETWORK SERVICE (a scheduled task with -UserId 'NT AUTHORITY\NETWORK SERVICE' works). A first attempt at that probe wrote its output to the admin's $env:TEMP — which NETWORK SERVICE cannot write — and so reported a false FAIL for a fix that had actually worked. Write probe output somewhere the service account can reach (C:\Windows\Temp).

Still open¶

Cause 1 (billing) — the 3 ubuntu-latest scanners cannot start. Account action only.
LocalRagLiveSmokeTests — needs a live Ollama the runner account cannot reach. Same account-scoping shape. It is a live-service dependency inside the standard suite, so Backend goes red whenever the local model is down. Options: set OLLAMA_HOST machine-wide; gate the test behind a trait so it is opt-in locally and skipped in CI; or accept the coupling. Not yet decided.